Webdriver implementation in Chicken Scheme

1. Dependencies

Dependency Description
srfi-34 Exception Handling
srfi-35 Exception Types
base64 decoding screenshot data
http-client API interaction
intarweb Supporting HTTP functionality
uri-common Supporting HTTP functionality
coops Object system
alist-lib Handling alists from JSON objects
medea JSON handling

2. Error Conditions

(define-condition-type &wd-exception &error wd-exception?
  (stacktrace wd-stacktrace)
  (data wd-data))

Every API error code (key "error" in the returned JSON data) gets its own condition type, prefixed by &. They all inherit from &wd-exception.

Name API Error Code
detached-shadow-root detached shadow root
element-click-intercepted element click intercepted
element-not-interactable element not interactable
insecure-certificate insecure certificate
invalid-argument invalid argument
invalid-cookie-domain invalid cookie domain
invalid-element-state invalid element state
invalid-selector invalid selector
invalid-session-id invalid session id
javascript-error javascript error
move-target-out-of-bounds move target out of bounds
no-such-alert no such alert
no-such-cookie no such cookie
no-such-element no such element
no-such-frame no such frame
no-such-shadow-root no such shadow root
no-such-window no such window
script-timeout script timeout
session-not-created session not created
stale-element-reference stale element reference
timeout timeout
unable-to-capture-screen unable to capture screen
unable-to-set-cookie unable to set cookie
unexpected-alert-open unexpected alert open
unknown-command unknown command
unknown-error unknown error
unknown-method unknown method
unsupported-operation unsupported operation

3. WebDriver

The core element of the library is the <WebDriver> class and its subclasses. The class has the following fields:

(define-class <WebDriver> ()
  ((browser #f)
   (active? #f)
   (browser-pid #f)
   (server #f)
   (port #f)
   (session-id #f)
   (prefs #f)
   (capabilities #f)))

The parent class provides a handful of methods, but does not implement all of them; some are the sole responsibility of the subclass. The launch method, on the other hand, bears shared responsibility. It sets a finalizer to ensure termination of the web driver process in case the class is disposed of with a still-open driver.

(define-method (launch #:after (instance <WebDriver>) options)
  (set-finalizer! instance (lambda (obj)
                             (when (slot-value instance 'active?)
                               (terminate instance)))))

(define-method (terminate (instance <WebDriver>))
  (terminate-session instance)
  (process-signal (slot-value instance 'browser-pid))
  (set! (slot-value instance 'browser-pid) #f)
  (set! (slot-value instance 'active?) #f))

(define-method (construct-capabilities (instance <WebDriver>) #!optional caps)
  (raise 'subclass-responsibility))

(define-method (postprocess-result (instance <WebDriver>) result)
  result)

Main initialization is done by calling the new-WebDriver procedure with the respective class name and optionally an alist of options.

(define (new-WebDriver browser #!optional options)
  (let ((instance (make browser)))
    (launch instance options)
    (sleep 1)
    instance))

3.1. Geckodriver

The Geckodriver is used to control Firefox.

(define-class <Gecko> (<WebDriver>)
  ((browser #:firefox)
   (server "127.0.0.1")
   (port 4444)))

(define-method (launch (instance <Gecko>) options)
  (let ((pid (process-run "geckodriver > /dev/null 2>&1")))
    (set! (slot-value instance 'browser-pid) pid)
    (set! (slot-value instance 'active?) #t)
    (set! (slot-value instance 'capabilities) options)))

The capabilities object for Geckodriver is of the form {"capabilities": {...}}. For more information on capabilities, see https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities.

(define-method (construct-capabilities (instance <Gecko>))
  (let ((caps (or (slot-value instance 'capabilities) (list))))
    `((capabilities . ,caps))))

Sometimes, Geckodriver returns the results of a command in a JSON object with the sole key "value". We have to correct that before returning the data to the user.

(define-method (postprocess-result (instance <Gecko>) result)
  (alist-ref/default result 'value result))

4. WebDriver API

4.1. Communication

Data is sent to the API via a central class method. For convenience, there is a send-with-session variant that automatically adds the session id.

(define-method (send (instance <WebDriver>) data uri method)
  (let* ((remote (string-append "http://" (slot-value instance 'server) ":" (->string (slot-value instance 'port)) "/"))
         (result (postprocess-result instance
                                     (with-input-from-request
                                      (make-request #:method method
                                                    #:uri (uri-reference (string-append remote uri))
                                                    #:headers (headers `((content-type application/json))))
                                      (if data (json->string data) "")
                                      read-json))))
    (if (and (list? result) (alist-ref/default result 'error #f))
        (wd-throw result)
        result)))

(define-method (send-with-session (instance <WebDriver>) data uri method)
  (send instance data (string-append "session/" (slot-value instance 'session-id) "/" uri) method))

4.2. Session management

Session management is very simple. There is just one method to initialize a new session. Everything else is handled automatically.

(define-method (initialize-session (instance <WebDriver>))
 (let ((result (send instance (construct-capabilities instance) "session" 'POST)))
   (set! (slot-value instance 'session-id) (alist-ref result 'sessionId))))
(define-method (terminate-session (instance <WebDriver>))
  (when (slot-value instance 'session-id)
    (send instance #f (string-append "session/" (slot-value instance 'session-id)) 'DELETE))
  (set! (slot-value instance 'session-id) #f))
-- testing session -----------------------------------------------------------
Initial state ........................................................ [ PASS]
Session id check ..................................................... [ PASS]
Session id after termination ......................................... [ PASS]
3 tests completed in 3.788 seconds.
3 out of 3 (100%) tests passed.
-- done testing session ------------------------------------------------------

4.3. API Access Methods

(define-method (set-url (instance <WebDriver>) url)
  (send-with-session instance `((url . ,url)) "url" 'POST))

(define-method (url (instance <WebDriver>))
  (send-with-session instance #f "url" 'GET))
-- testing url ---------------------------------------------------------------
Initial state ........................................................ [ PASS]
Navigating to the first website ...................................... [ PASS]
2 tests completed in 5.247 seconds.
2 out of 2 (100%) tests passed.
-- done testing url ----------------------------------------------------------
(define-method (back (instance <WebDriver>))
  (send-with-session instance #f "back" 'POST))
(define-method (forward (instance <WebDriver>))
  (send-with-session instance #f "forward" 'POST))
(define-method (refresh (instance <WebDriver>))
  (send-with-session instance #f "refresh" 'POST))
(define-method (title (instance <WebDriver>))
  (send-with-session instance #f "title" 'GET))
(define-method (status (instance <WebDriver>))
  (send-with-session instance #f "status" 'GET))
(define-method (source (instance <WebDriver>))
  (send-with-session instance #f "source" 'GET))
(define-method (screenshot (instance <WebDriver>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))
(define-method (print-page (instance <WebDriver>))
  (send-with-session instance #f "print" 'POST))
(define-method (execute-async (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/async" 'POST))
(define-method (execute-sync (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/sync" 'POST))

4.4. Timeouts

The following timeouts are defined:

  • script: defaults to 30'000, specifies when to interrupt a script that is being evaluated. A nil value implies that scripts should never be interrupted, but instead run indefinitely.
  • pageLoad: defaults to 300'000, provides the timeout limit used to interrupt an explicit navigation attempt.
  • implicit: defaults to 0, specifies a time to wait for the element location strategy to complete when locating an element.
(define-class <WDTimeouts> ()
  ((script 30000)
   (pageLoad 300000)
   (implicit 0)))
(define-method (extract (instance <WDTimeouts>))
  `((script . ,(slot-value instance 'script))
    (pageLoad . ,(slot-value instance 'pageLoad))
    (implicit . ,(slot-value instance 'implicit))))

4.4.1. Setting and getting timeouts

(define-method (set-timeouts (instance <WebDriver>) (timeouts <WDTimeouts>))
  (send-with-session instance (extract timeouts) "timeouts" 'POST))

(define-method (timeouts (instance <WebDriver>))
  (let ((result (send-with-session instance #f "timeouts" 'GET)))
    (make <WDTimeouts>
      'script (alist-ref result 'script)
      'pageLoad (alist-ref result 'pageLoad)
      'implicit (alist-ref result 'implicit))))

4.5. Elements

4.5.1. Element Class

(define-class <WDElement> ()
  ((driver #f)
   (element #f)))
(define-method (send-with-session (instance <WDElement>) data uri method)
  (send-with-session (slot-value instance 'driver) data
                     (string-append "element/" (slot-value instance 'element) "/" uri)
                     method))

4.5.2. Finding Elements

  1. Location Strategies
    (define css-selector "css selector")
    (define link-text "link text")
    (define partial-link-text "partial link text")
    (define tag-name "tag name")
    (define xpath "xpath")
    
  2. Accessor Methods
    (define-method (find-element (instance <WebDriver>) strategy selector)
      (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
        (make <WDElement> 'driver instance 'element (car (alist-values result)))
        element))
    
    (define-method (find-elements (instance <WebDriver>) strategy selector)
      (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
        (map
         (lambda (elem)
           (make <WDElement> 'driver instance 'element (car (alist-values elem))))
         result)))
    
    (define-method (find-element (instance <WDElement>) strategy selector)
      (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
        (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values result)))
        element))
    
    (define-method (find-elements (instance <WDElement>) strategy selector)
      (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
        (map
         (lambda (elem)
           (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values elem))))
         result)))
    

4.5.3. Working with Elements

(define-method (attribute (instance <WDElement>) attribute)
  (let ((result (send-with-session instance #f
                                   (string-append "attribute/" attribute)
                                   'GET)))
    (if (equal? "true" result)
        #t
        result)))
(define-method (property (instance <WDElement>) property)
  (send-with-session instance #f (string-append "property/" property) 'GET))
(define-method (clear (instance <WDElement>))
  (send-with-session instance #f "clear" 'POST))
(define-method (click (instance <WDElement>))
  (send-with-session instance #f "click" 'POST))
(define-method (computed-label (instance <WDElement>))
  (send-with-session instance #f "computedlabel" 'GET))
(define-method (computed-role (instance <WDElement>))
  (send-with-session instance #f "computedrole" 'GET))
(define-method (enabled? (instance <WDElement>))
  (send-with-session instance #f "enabled" 'GET))
(define-method (selected? (instance <WDElement>))
  (send-with-session instance #f "selected" 'GET))
(define-method (name (instance <WDElement>))
  (send-with-session instance #f "name" 'GET))
(define-method (rect (instance <WDElement>))
  (send-with-session instance #f "rect" 'GET))
(define-method (screenshot (instance <WDElement>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))
(define-method (text (instance <WDElement>))
  (send-with-session instance #f "text" 'GET))
(define-method (set-value (instance <WDElement>) value)
  (send-with-session instance `((text . ,value)) "value" 'POST))

5. About This Egg

5.1. Source

The source is available at https://fossil.lyrion.ch/chicken-webdriver.

5.2. Author

Daniel Ziltener

5.3. Version History

0.5 Initial Release

5.4. License

Copyright (C) 2023 Daniel Ziltener

All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the <organization> nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Author: Daniel Ziltener

Created: 2023-04-13 Do 17:01

Validate