#+title: Webdriver implementation in Chicken Scheme
#+author: Daniel Ziltener
#+property: header-args:scheme :session *chicken* :comments both


* Helpers :noexport:
:PROPERTIES:
:header-args:scheme: :prologue "(import (chicken string))"
:END:
** Strip garbage from test results
#+name: test-post
#+begin_src scheme :var input='() :results output
(for-each (lambda (str)
            (or (substring=? str ";")
                (substring=? str "Note")
                (print str)))
          (string-split input "\n"))
#+end_src

** Prepare in-line testing
#+name: prep-test
#+begin_src scheme :noweb yes :tangle tests/run.scm :results silent
(import r7rs
        test
        (chicken base)
        (chicken string)
        (chicken process)
        (chicken gc)
        <<dependencies-for-imports()>>
)
#+end_src

* Dependencies

#+name: dependencies
| Dependency  | Description                       |
|-------------+-----------------------------------|
| srfi-34     | Exception Handling                |
| srfi-35     | Exception Types                   |
| base64      | decoding screenshot data          |
| http-client | API interaction                   |
| intarweb    | Supporting HTTP functionality     |
| uri-common  | Supporting HTTP functionality     |
| coops       | Object system                     |
| alist-lib   | Handling alists from JSON objects |
| medea       | JSON handling                     |

#+name: dependencies-for-egg
#+begin_src emacs-lisp :var tbl=dependencies :colnames yes :results raw :exports none
(mapconcat (lambda (row) (car row)) tbl " ")
#+end_src

#+name: dependencies-for-imports
#+begin_src emacs-lisp :var tbl=dependencies :colnames yes :results raw :exports none
(mapconcat (lambda (row) (concat (car row) "\t\t;;" (cadr row)))
           tbl "\n")
#+end_src

#+begin_src scheme :noweb yes :tangle webdriver.scm :exports none
(import r7rs)
(define-library (webdriver)
  (import (scheme base))
  (export <WDElement>)
  (include "webdriver-impl.scm"))
#+end_src

#+begin_src scheme :noweb yes :tangle webdriver-impl.scm :exports none
(import r7rs
        (chicken base)
        (chicken string)
        (chicken process)
        (chicken gc)
        <<dependencies-for-imports()>>
        )
#+end_src

#+begin_src scheme :tangle tests/run.scm :exports none :mkdirp yes
(include-relative "../webdriver-impl.scm")
#+end_src

* Error Conditions

#+name: wd-exception
#+begin_src scheme :tangle webdriver-impl.scm
(define-condition-type &wd-exception &error wd-exception?
  (stacktrace wd-stacktrace)
  (data wd-data))
#+end_src

Every API error code (key "error" in the returned JSON data) gets its own condition type, prefixed by ~&~. They all inherit from ~&wd-exception~.

#+name: error-code-table
| Name                      | API Error Code            |
|---------------------------+---------------------------|
| detached-shadow-root      | detached shadow root      |
| element-click-intercepted | element click intercepted |
| element-not-interactable  | element not interactable  |
| insecure-certificate      | insecure certificate      |
| invalid-argument          | invalid argument          |
| invalid-cookie-domain     | invalid cookie domain     |
| invalid-element-state     | invalid element state     |
| invalid-selector          | invalid selector          |
| invalid-session-id        | invalid session id        |
| javascript-error          | javascript error          |
| move-target-out-of-bounds | move target out of bounds |
| no-such-alert             | no such alert             |
| no-such-cookie            | no such cookie            |
| no-such-element           | no such element           |
| no-such-frame             | no such frame             |
| no-such-shadow-root       | no such shadow root       |
| no-such-window            | no such window            |
| script-timeout            | script timeout            |
| session-not-created       | session not created       |
| stale-element-reference   | stale element reference   |
| timeout                   | timeout                   |
| unable-to-capture-screen  | unable to capture screen  |
| unable-to-set-cookie      | unable to set cookie      |
| unexpected-alert-open     | unexpected alert open     |
| unknown-command           | unknown command           |
| unknown-error             | unknown error             |
| unknown-method            | unknown method            |
| unsupported-operation     | unsupported operation     |

#+name: define-conditions
#+begin_src emacs-lisp :var src=error-code-table :exports none :results raw
(mapconcat
 (lambda (row)
   (let ((replace `((?n . ,(cl-first row))
                    (?c . ,(cl-second row)))))
     (format-spec "(define-condition-type &%n &wd-exception %n?)" replace)))
 src "\n")
#+end_src

#+name: define-condition-thrower
#+begin_src emacs-lisp :var src=error-code-table :exports none :results raw
(concat "(define (wd-throw data)\n"
        "  (case (alist-ref data 'error)\n"
        (mapconcat
         (lambda (row)
           (let ((replace `((?n . ,(cl-first row))
                            (?c . ,(cl-second row)))))
             (format-spec "    ((\"%c\") (raise (make-condition &%n (alist-ref data 'stacktrace) data)))" replace)))
         src "\n")
        "\n"
        "    (else (raise (make-condition &wd-exception (alist-ref data 'stacktrace) data)))\n"
        "  )\n"
        ")")
#+end_src

#+name: conditions
#+begin_src scheme :tangle webdriver-impl.scm :noweb yes :exports none
<<define-conditions()>>

<<define-condition-thrower()>>
#+end_src

* WebDriver

The core element of the library is the ~<WebDriver>~ class and its subclasses. The class has the following fields:

#+name: webdriver-class
#+begin_src scheme :tangle webdriver-impl.scm
(define-class <WebDriver> ()
  ((browser #f)
   (active? #f)
   (browser-pid #f)
   (server #f)
   (port #f)
   (session-id #f)
   (prefs #f)
   (capabilities #f)))
#+end_src

The parent class provides a handful of methods, but does not implement all of them; some are the sole responsibility of the subclass. The ~launch~ method, on the other hand, bears shared responsibility. It sets a finalizer to ensure termination of the web driver process in case the class is disposed of with a still-open driver.

#+name: webdriver-basics
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (launch #:after (instance <WebDriver>) options)
  (set-finalizer! instance (lambda (obj)
                             (when (slot-value instance 'active?)
                               (terminate instance)))))

(define-method (terminate (instance <WebDriver>))
  (terminate-session instance)
  (process-signal (slot-value instance 'browser-pid))
  (set! (slot-value instance 'browser-pid) #f)
  (set! (slot-value instance 'active?) #f))

(define-method (construct-capabilities (instance <WebDriver>) #!optional caps)
  (raise 'subclass-responsibility))

(define-method (postprocess-result (instance <WebDriver>) result)
  result)
#+end_src

Main initialization is done by calling the ~new-WebDriver~ procedure with the respective class name and optionally an alist of options.

#+name: webdriver-init
#+begin_src scheme :tangle webdriver-impl.scm
(define (new-WebDriver browser #!optional options)
  (let ((instance (make browser)))
    (launch instance options)
    (sleep 1)
    instance))
#+end_src

** Geckodriver

The Geckodriver is used to control Firefox.

#+name: geckodriver-basic
#+begin_src scheme :tangle webdriver-impl.scm
(define-class <Gecko> (<WebDriver>)
  ((browser #:firefox)
   (server "127.0.0.1")
   (port 4444)))

(define-method (launch (instance <Gecko>) options)
  (let ((pid (process-run "geckodriver > /dev/null 2>&1")))
    (set! (slot-value instance 'browser-pid) pid)
    (set! (slot-value instance 'active?) #t)
    (set! (slot-value instance 'capabilities) options)))
#+end_src

The capabilities object for Geckodriver is of the form ={"capabilities": {...}}=.
For more information on capabilities, see https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities.

#+name: geckodriver-capabilities
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (construct-capabilities (instance <Gecko>))
  (let ((caps (or (slot-value instance 'capabilities) (list))))
    `((capabilities . ,caps))))
#+end_src

Sometimes, Geckodriver returns the results of a command in a JSON object with the sole key ="value"=. We have to correct that before returning the data to the user.

#+name: geckodriver-postprocess
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (postprocess-result (instance <Gecko>) result)
  (alist-ref/default result 'value result))
#+end_src

#+name: prep-geckodriver-test
#+begin_src scheme :tangle tests/run.scm :noweb strip-tangle :exports none :post test-post(input=*this*) :results output
<<prep-test>>
<<wd-exception>>
<<conditions>>
<<webdriver-class>>
<<webdriver-basics>>
<<webdriver-init>>
<<geckodriver-basic>>
<<geckodriver-capabilities>>
<<geckodriver-postprocess>>
<<wd-send>>
<<wd-init-session>>
<<wd-term-session>>
#+end_src

* WebDriver API

** Communication

Data is sent to the API via a central class method. For convenience, there is a ~send-with-session~ variant that automatically adds the session id.

#+name: wd-send
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (send (instance <WebDriver>) data uri method)
  (let* ((remote (string-append "http://" (slot-value instance 'server) ":" (->string (slot-value instance 'port)) "/"))
         (result (postprocess-result instance
                                     (with-input-from-request
                                      (make-request #:method method
                                                    #:uri (uri-reference (string-append remote uri))
                                                    #:headers (headers `((content-type application/json))))
                                      (if data (json->string data) "")
                                      read-json))))
    (if (and (list? result) (alist-ref/default result 'error #f))
        (wd-throw result)
        result)))

(define-method (send-with-session (instance <WebDriver>) data uri method)
  (send instance data (string-append "session/" (slot-value instance 'session-id) "/" uri) method))
#+end_src

** Session management

Session management is very simple. There is just one method to initialize a new session. Everything else is handled automatically.

#+name: wd-init-session
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (initialize-session (instance <WebDriver>))
 (let ((result (send instance (construct-capabilities instance) "session" 'POST)))
   (set! (slot-value instance 'session-id) (alist-ref result 'sessionId))))
#+end_src

#+name: wd-term-session
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (terminate-session (instance <WebDriver>))
  (when (slot-value instance 'session-id)
    (send instance #f (string-append "session/" (slot-value instance 'session-id)) 'DELETE))
  (set! (slot-value instance 'session-id) #f))
#+end_src

#+name: wd-session-test
#+begin_src scheme :tangle tests/run.scm :noweb strip-tangle :exports results :post test-post(input=*this*) :results output
<<prep-geckodriver-test>>
(test-group "session"
  (let ((browser (new-WebDriver <Gecko>)))
    (test "Initial state" #f (slot-value browser 'session-id))
    (test-assert "Session id check" (string? (begin (initialize-session browser) (slot-value browser 'session-id))))
    (test-assert "Session id after termination" (eq? #f (begin (terminate-session browser) (slot-value browser 'session-id))))
    (terminate browser)))
#+end_src

#+RESULTS: wd-session-test
: -- testing session -----------------------------------------------------------
: Initial state ........................................................ [ PASS]
: Session id check ..................................................... [ PASS]
: Session id after termination ......................................... [ PASS]
: 3 tests completed in 3.788 seconds.
: 3 out of 3 (100%) tests passed.
: -- done testing session ------------------------------------------------------

** API Access Methods

#+name: wd-url
#+begin_src scheme :tangle webdriver-impl.scm
(define-method (set-url (instance <WebDriver>) url)
  (send-with-session instance `((url . ,url)) "url" 'POST))

(define-method (url (instance <WebDriver>))
  (send-with-session instance #f "url" 'GET))
#+end_src

#+name: wd-url-test
#+begin_src scheme :tangle tests/run.scm :noweb strip-tangle :exports results :post test-post(input=*this*) :results output
<<prep-geckodriver-test>>
<<wd-url>>
(test-group "url"
  (let ((browser (new-WebDriver <Gecko>)))
    (test "Initial state" #f (slot-value browser 'session-id))
    (test "Navigating to the first website" "http://info.cern.ch/hypertext/WWW/TheProject.html"
          (begin (initialize-session browser)
                 (set-url browser "http://info.cern.ch/hypertext/WWW/TheProject.html")
                 (url browser)))
    (terminate browser)))
#+end_src

#+RESULTS: wd-url-test
: -- testing url ---------------------------------------------------------------
: Initial state ........................................................ [ PASS]
: Navigating to the first website ...................................... [ PASS]
: 2 tests completed in 5.247 seconds.
: 2 out of 2 (100%) tests passed.
: -- done testing url ----------------------------------------------------------

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (back (instance <WebDriver>))
  (send-with-session instance #f "back" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (forward (instance <WebDriver>))
  (send-with-session instance #f "forward" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (refresh (instance <WebDriver>))
  (send-with-session instance #f "refresh" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (title (instance <WebDriver>))
  (send-with-session instance #f "title" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (status (instance <WebDriver>))
  (send-with-session instance #f "status" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (source (instance <WebDriver>))
  (send-with-session instance #f "source" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (screenshot (instance <WebDriver>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (print-page (instance <WebDriver>))
  (send-with-session instance #f "print" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (execute-async (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/async" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (execute-sync (instance <WebDriver>) script args)
  (send-with-session instance `((script . ,script) (args . ,args)) "execute/sync" 'POST))
#+end_src

** Timeouts

The following timeouts are defined:

- =script=: defaults to 30'000, specifies when to interrupt a script that is being evaluated. A nil value implies that scripts should never be interrupted, but instead run indefinitely.
- =pageLoad=: defaults to 300'000, provides the timeout limit used to interrupt an explicit navigation attempt.
- =implicit=: defaults to 0, specifies a time to wait for the element location strategy to complete when locating an element.

#+begin_src scheme :tangle webdriver-impl.scm
(define-class <WDTimeouts> ()
  ((script 30000)
   (pageLoad 300000)
   (implicit 0)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (extract (instance <WDTimeouts>))
  `((script . ,(slot-value instance 'script))
    (pageLoad . ,(slot-value instance 'pageLoad))
    (implicit . ,(slot-value instance 'implicit))))
#+end_src

*** Setting and getting timeouts

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (set-timeouts (instance <WebDriver>) (timeouts <WDTimeouts>))
  (send-with-session instance (extract timeouts) "timeouts" 'POST))

(define-method (timeouts (instance <WebDriver>))
  (let ((result (send-with-session instance #f "timeouts" 'GET)))
    (make <WDTimeouts>
      'script (alist-ref result 'script)
      'pageLoad (alist-ref result 'pageLoad)
      'implicit (alist-ref result 'implicit))))
#+end_src

** Elements

*** Element Class

#+begin_src scheme :tangle webdriver-impl.scm
(define-class <WDElement> ()
  ((driver #f)
   (element #f)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (send-with-session (instance <WDElement>) data uri method)
  (send-with-session (slot-value instance 'driver) data
                     (string-append "element/" (slot-value instance 'element) "/" uri)
                     method))
#+end_src

*** Finding Elements

**** Location Strategies

#+begin_src scheme :tangle webdriver-impl.scm
(define css-selector "css selector")
(define link-text "link text")
(define partial-link-text "partial link text")
(define tag-name "tag name")
(define xpath "xpath")
#+end_src

**** Accessor Methods

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (find-element (instance <WebDriver>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
    (make <WDElement> 'driver instance 'element (car (alist-values result)))
    element))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (find-elements (instance <WebDriver>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
    (map
     (lambda (elem)
       (make <WDElement> 'driver instance 'element (car (alist-values elem))))
     result)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (find-element (instance <WDElement>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "element" 'POST)))
    (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values result)))
    element))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (find-elements (instance <WDElement>) strategy selector)
  (let ((result (send-with-session instance `((using . ,strategy) (value . ,selector)) "elements" 'POST)))
    (map
     (lambda (elem)
       (make <WDElement> 'driver (slot-value instance 'driver) 'element (car (alist-values elem))))
     result)))
#+end_src

*** Working with Elements

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (attribute (instance <WDElement>) attribute)
  (let ((result (send-with-session instance #f
                                   (string-append "attribute/" attribute)
                                   'GET)))
    (if (equal? "true" result)
        #t
        result)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (property (instance <WDElement>) property)
  (send-with-session instance #f (string-append "property/" property) 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (clear (instance <WDElement>))
  (send-with-session instance #f "clear" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (click (instance <WDElement>))
  (send-with-session instance #f "click" 'POST))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (computed-label (instance <WDElement>))
  (send-with-session instance #f "computedlabel" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (computed-role (instance <WDElement>))
  (send-with-session instance #f "computedrole" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (enabled? (instance <WDElement>))
  (send-with-session instance #f "enabled" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (selected? (instance <WDElement>))
  (send-with-session instance #f "selected" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (name (instance <WDElement>))
  (send-with-session instance #f "name" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (rect (instance <WDElement>))
  (send-with-session instance #f "rect" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (screenshot (instance <WDElement>))
  (base64-decode (send-with-session instance #f "screenshot" 'GET)))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (text (instance <WDElement>))
  (send-with-session instance #f "text" 'GET))
#+end_src

#+begin_src scheme :tangle webdriver-impl.scm
(define-method (set-value (instance <WDElement>) value)
  (send-with-session instance `((text . ,value)) "value" 'POST))
#+end_src

* About This Egg

#+begin_src scheme :noweb yes :tangle webdriver.egg :exports none
;; -*- scheme -*-
((author "Daniel Ziltener")
 (synopsis "A WebDriver API implementation for Chicken")
 (category web)
 (license "BSD")
 (version <<latest-release()>>)
 (dependencies r7rs <<dependencies-for-egg()>>)
 (test-dependencies test)

 (components
  (extension webdriver
             (csc-options "-X" "r7rs" "-R" "r7rs" "-sJ"))))
#+end_src

#+begin_src scheme :tangle tests/run.scm :exports none
(test-exit)
#+end_src

** Source

The source is available at [[https://fossil.lyrion.ch/chicken-webdriver]].

** Author

Daniel Ziltener

** Version History

#+name: version-history
| 0.5 | Initial Release |

#+name: gen-releases
#+begin_src emacs-lisp :var vers=version-history :results raw :exports none
(mapconcat (lambda (row) (concat "(release \"" (number-to-string (car row)) "\") ;; " (cadr row)))
           vers "\n")
#+end_src

#+name: latest-release
#+begin_src emacs-lisp :var vers=version-history :exports none :results code
(number-to-string (caar vers))
#+end_src

#+begin_src scheme :noweb yes :tangle webdriver.release-info :exports none
;; -*- scheme -*-
(repo fossil "https://fossil.lyrion.ch/chicken-webdriver")
(uri targz "https://fossil.lyrion.ch/chicken-webdriver/tarball/{egg-release}/chicken-webdriver-{egg-release}.tar.gz")
<<gen-releases()>>
#+end_src

** License

#+begin_src fundamental :tangle LICENSE
Copyright (C) 2023 Daniel Ziltener

All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the <organization> nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#+end_src