Webdriver implementation in Chicken Scheme

1. Dependencies
2. Error Conditions
3. WebDriver
- 3.1. Geckodriver
4. WebDriver API

1. Dependencies

Dependency	Description
srfi-34	Exception Handling
srfi-35	Exception Types
base64	decoding screenshot data
http-client	API interaction
intarweb	Supporting HTTP functionality
uri-common	Supporting HTTP functionality
coops	Object system
alist-lib	Handling alists from JSON objects
medea	JSON handling

2. Error Conditions

(define-condition-type &wd-exception &error wd-exception?
  (stacktrace wd-stacktrace)
  (data wd-data))

Every API error code (key “error” in the returned JSON data) gets its own condition type, prefixed by &. They all inherit from &wd-exception.

Name	API Error Code
detached-shadow-root	detached shadow root
element-click-intercepted	element click intercepted
element-not-interactable	element not interactable
insecure-certificate	insecure certificate
invalid-argument	invalid argument
invalid-cookie-domain	invalid cookie domain
invalid-element-state	invalid element state
invalid-selector	invalid selector
invalid-session-id	invalid session id
javascript-error	javascript error
move-target-out-of-bounds	move target out of bounds
no-such-alert	no such alert
no-such-cookie	no such cookie
no-such-element	no such element
no-such-frame	no such frame
no-such-shadow-root	no such shadow root
no-such-window	no such window
script-timeout	script timeout
session-not-created	session not created
stale-element-reference	stale element reference
timeout	timeout
unable-to-capture-screen	unable to capture screen
unable-to-set-cookie	unable to set cookie
unexpected-alert-open	unexpected alert open
unknown-command	unknown command
unknown-error	unknown error
unknown-method	unknown method
unsupported-operation	unsupported operation

3. WebDriver

The core element of the library is the <WebDriver> class and its subclasses. The class has the following fields:

(define-class <WebDriver> ()
  ((browser #f)
   (active? #f)
   (browser-pid #f)
   (server #f)
   (port #f)
   (session-id #f)
   (prefs #f)
   (capabilities #f)))

The parent class provides a handful of methods, but does not implement all of them; some are the sole responsibility of the subclass. The launch method, on the other hand, bears shared responsibility. It sets a finalizer to ensure termination of the web driver process in case the class is disposed of with a still-open driver.

(define-method (launch #:after (instance <WebDriver>) options)
  (set-finalizer! instance (lambda (obj)
                             (when (slot-value instance 'active?)
                               (terminate instance)))))

(define-method (terminate (instance <WebDriver>))
  (terminate-session instance)
  (process-signal (slot-value instance 'browser-pid))
  (set! (slot-value instance 'browser-pid) #f)
  (set! (slot-value instance 'active?) #f))

(define-method (construct-capabilities (instance <WebDriver>) #!optional caps)
  (raise 'subclass-responsibility))

(define-method (postprocess-result (instance <WebDriver>) result)
  result)

Main initialization is done by calling the new-WebDriver procedure with the respective class name and optionally an alist of options.

(define (new-WebDriver browser #!optional options)
  (let ((instance (make browser)))
    (launch instance options)
    (sleep 1)
    instance))

3.1. Geckodriver

The Geckodriver is used to control Firefox.

(define-class <Gecko> (<WebDriver>)
  ((browser #:firefox)
   (server "127.0.0.1")
   (port 4444)))

(define-method (launch (instance <Gecko>) options)
  (let ((pid (process-run "geckodriver > /dev/null 2>&1")))
    (set! (slot-value instance 'browser-pid) pid)
    (set! (slot-value instance 'active?) #t)
    (set! (slot-value instance 'capabilities) options)))

The capabilities object for Geckodriver is of the form {"capabilities": {...}}. For more information on capabilities, see https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities.

(define-method (construct-capabilities (instance <Gecko>))
  (let ((caps (or (slot-value instance 'capabilities) (list))))
    `((capabilities . ,caps))))

Sometimes, Geckodriver returns the results of a command in a JSON object with the sole key "value". We have to correct that before returning the data to the user.

(define-method (postprocess-result (instance <Gecko>) result)
  (alist-ref/default result 'value result))

4. WebDriver API

4.1. Communication

Data is sent to the API via a central class method. For convenience, there is a send-with-session variant that automatically adds the session id.

(define-method (send (instance <WebDriver>) data uri method)
  (let* ((remote (string-append "http://" (slot-value instance 'server) ":" (->string (slot-value instance 'port)) "/"))
         (result (postprocess-result instance
                                     (with-input-from-request
                                      (make-request #:method method
                                                    #:uri (uri-reference (string-append remote uri))
                                                    #:headers (headers `((content-type application/json))))
                                      (if data (json->string data) "")
                                      read-json))))
    (if (and (list? result) (alist-ref/default result 'error #f))
        (wd-throw result)
        result)))

(define-method (send-with-session (instance <WebDriver>) data uri method)
  (send instance data (string-append "session/" (slot-value instance 'session-id) "/" uri) method))

4.2. Session management

Session management is very simple. There is just one method to initialize a new session. Everything else is handled automatically.

(define-method (initialize-session (instance <WebDriver>))
 (let ((result (send instance (construct-capabilities instance) "session" 'POST)))
   (set! (slot-value instance 'session-id) (alist-ref result 'sessionId))))

(define-method (terminate-session (instance <WebDriver>))
  (when (slot-value instance 'session-id)
    (send instance #f (string-append "session/" (slot-value instance 'session-id)) 'DELETE))
  (set! (slot-value instance 'session-id) #f))

-- testing session -----------------------------------------------------------
Initial state ........................................................ [ PASS]
Session id check ..................................................... [ PASS]
Session id after termination ......................................... [ PASS]
3 tests completed in 4.952 seconds.
3 out of 3 (100%) tests passed.
-- done testing session ------------------------------------------------------

4.3. API Access Methods

(define-method (set-url (instance <WebDriver>) url)
  (send-with-session instance `((url . ,url)) "url" 'POST))

(define-method (url (instance <WebDriver>))
  (send-with-session instance #f "url" 'GET))

-- testing url ---------------------------------------------------------------
Initial state ........................................................ [ PASS]
Navigating to the first website ...................................... [ PASS]
2 tests completed in 5.471 seconds.
2 out of 2 (100%) tests passed.
-- done testing url ----------------------------------------------------------

(define-method (back (instance <WebDriver>))
  (send-with-session instance #f "back" 'POST))

(define-method (forward (instance <WebDriver>))
  (send-with-session instance #f "forward" 'POST))

(define-method (forward (instance <WebDriver>))
  (send-with-session instance #f "refresh" 'POST))

(define-method (title (instance <WebDriver>))
  (send-with-session instance #f "title" 'GET))

(define-method (status (instance <WebDriver>))
  (send-with-session instance #f "status" 'GET))

(define-method (source (instance <WebDriver>))
  (send-with-session instance #f "source" 'GET))

(define-method (screenshot (instance <WebDriver>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))

(define-method (print-page (instance <WebDriver>))
  (send-with-session instance #f "print" 'POST))

(define-method (execute-async (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/async" 'POST))

(define-method (execute-sync (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/sync" 'POST))

4.4. Timeouts

The following timeouts are defined:

script: defaults to 30’000, specifies when to interrupt a script that is being evaluated. A nil value implies that scripts should never be interrupted, but instead run indefinitely.
pageLoad: defaults to 300’000, provides the timeout limit used to interrupt an explicit navigation attempt.
implicit: defaults to 0, specifies a time to wait for the element location strategy to complete when locating an element.

(define-class <WDTimeouts> ()
  ((script 30000)
   (pageLoad 300000)
   (implicit 0)))

(define-method (extract (instance <WDTimeouts>))
  `((script . ,(slot-value instance 'script))
    (pageLoad . ,(slot-value instance 'pageLoad))
    (implicit . ,(slot-value instance 'implicit))))

4.4.1. Setting and getting timeouts

(define-method (set-timeouts (instance <WebDriver>) (timeouts <WDTimeouts>))
  (send-with-session instance (extract timeouts) "timeouts" 'POST))

(define-method (timeouts (instance <WebDriver>))
  (let ((result (send-with-session instance #f "timeouts" 'GET)))
    (make <WDTimeouts>
      'script (alist-ref result 'script)
      'pageLoad (alist-ref result 'pageLoad)
      'implicit (alist-ref result 'implicit))))

4.5. Elements

4.5.1. Element Class

(define-class <WDElement> ()
  ((driver #f)
   (element #f)))

(define-method (send-with-session (instance <WDElement>) data uri method)
  (send-with-session (slot-value instance 'driver) data
                     (string-append "element/" (slot-value instance 'element) "/" uri)
                     method))

4.5.2. Finding Elements

Location Strategies

(define css-selector "css selector")
(define link-text "link text")
(define partial-link-text "partial link text")
(define tag-name "tag name")
(define xpath "xpath")

Accessor Methods

(define-method (find-element (instance <WebDriver>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
    (make <WDElement> 'driver instance 'element (car (alist-values result)))
    element))

(define-method (find-elements (instance <WebDriver>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
    (map
     (lambda (elem)
       (make <WDElement> 'driver instance 'element (car (alist-values elem))))
     result)))

(define-method (find-element (instance <WDElement>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
    (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values result)))
    element))

(define-method (find-elements (instance <WDElement>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
    (map
     (lambda (elem)
       (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values elem))))
     result)))

4.5.3. Working with Elements

(define-method (attribute (instance <WDElement>) attribute)
  (let ((result (send-with-session instance #f
                                   (string-append "attribute/" attribute)
                                   'GET)))
    (if (equal? "true" result)
        #t
        result)))

(define-method (property (instance <WDElement>) property)
  (send-with-session instance #f (string-append "property/" property) 'GET))

(define-method (clear (instance <WDElement>))
  (send-with-session instance #f "clear" 'POST))

(define-method (click (instance <WDElement>))
  (send-with-session instance #f "click" 'POST))

(define-method (computed-label (instance <WDElement>))
  (send-with-session instance #f "computedlabel" 'GET))

(define-method (computed-role (instance <WDElement>))
  (send-with-session instance #f "computedrole" 'GET))

(define-method (enabled? (instance <WDElement>))
  (send-with-session instance #f "enabled" 'GET))

(define-method (selected? (instance <WDElement>))
  (send-with-session instance #f "selected" 'GET))

(define-method (name (instance <WDElement>))
  (send-with-session instance #f "name" 'GET))

(define-method (rect (instance <WDElement>))
  (send-with-session instance #f "rect" 'GET))

(define-method (screenshot (instance <WDElement>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))

(define-method (text (instance <WDElement>))
  (send-with-session instance #f "text" 'GET))

(define-method (set-value (instance <WDElement>) value)
  (send-with-session instance `((text . ,value)) "value" 'POST))