What is cMQL

  • query and data processing language for MongoDB
  • generates MQL that can be used as standalone commands, or as arguments in driver methods
  • main characteristics
    • up to 3x less code
    • simple structure of code
    • simple notation
  • () for code , {} for data
    like MQL with () see also
  • portable queries in both cmql-js and cmql-java
  • with the same performance

cMQL in order to archive the above goals, is made with Clojure
It can be used as a tool to generate MQL or to call cMQL code directly.

Learning guide

Examples

The easiest way to start is to use it online at cMQL-Play It has > 100 examples, type on search box the MQL operator that you want to search for, and results will appear sorted by query size.

StackOverflow was used to create the examples, to see the question press on the ExLink

cMQL-Play is an online cMQL to MQL generator, providing a subset of cMQL (2 commands aggregate/update) Bellow is 4 examples just to show how cMQL looks like.

Example1

(q (= :bedrooms 1)
(= :country.code "GR")
(group {:_id :stars}
{:average-price (avg :price)})
(sort :average-price)
(limit 1))

cMQL improvements

  • used common operators names for example = not "$eq"
  • replaced {} with () for code
  • simpler notation :field for references, instead of $ ""
  • less verbose
  • auto-add stages like match

Result is more clear code, clear structure, significant less code.
(95 less non-whitespace characters and ~2x less code)

Example2

(q :people.workers
(< :salary 1000)
(> :years 1)
{:children (filter (fn [:child.] (< :child.age. 15)) :all-children)}
(>= :children 2)
{:bonus (reduce (fn [:total. :child.]
(cond (< :child.age. 5) (+ :total. 100)
(< :child.age. 10) (+ :total. 50)
:else (+ :total. 20)))
0
:children)}
[:!id :name {:new-salary (+ :salary (if- (> :bonus 200) 200 :bonus))}]
(sort :!new-salary))

cMQL improvements

  • the ones that are noted the previous example including
  • auto added those common used stages
    • () -> for filters
    • {} -> for addFields
    • [] -> for project
  • mql variable notation
    • :child. = $$child
    • :child.age.="$$child.age"
  • sort and project simpler with ! for the opposite value
  • filter/map/reduce minimalistic with simple programming structure
    (filter (fn [:child.] ...) :all-children)
    (reduce (fn [:total. :child.] ...) :children)
    Alternative if we didn't want to use extra variables and $let in reduce
    (filter (fn [:this.] ...) :all-children)
    (reduce (fn [:value. :this.] ...) :children)

Example 3

cMQL is about less code, simple notation, good structure.
Here ~3x less code, 208 characters less, with simpler notation.

But here the most important is the structure. cMQL makes big and complicated queries easy to read and write.

(q {:s
(let [:a. (reduce (fn [:sum. :n.] (+ :sum. :n.))
0
(filter (fn [:w.] (some? :w.))
(map (fn [:v.] (+ :v. 1)) :myarray)))]
:a.)})

Example 4

Overview query of many aggregate operators.

The difference is clear, in all levels

  • size of code (+1330 more characters)
  • lines of code (+ 60 more lines)
  • notation
  • structure
(q (= :a (div (* :b (pow 20 2)) 10))
(= :b (not (or (< :a 1) (> :a 2))))
(= :c (trim (upper-case (subs :f 0 5))))
(= :d (count-str (str "a" :c "c")))
(or (exists? :c) (string? :d) (array? :b) (object? :c) (long? :d))
{
:a (map (fn [:v.] (+ :v. 1)) :myarray)
:b (filter (fn [:w.] (some? :w.)) :myarray)
:c (reduce (fn [:sum. :n.] (+ :sum. :n.)) 0 :myarray)
:d (concat :myarray (conj :myarray 10) (take 2 10 :myarray))
:e (get (take 10 (reverse (ziparray :x :y :z))) 2)
:f (contains? (difference (union :x :y (intersection :w :z)) [1 2 3]) 4)
:g (all-true? :z)
:h (empty? :z)
}
[
{:j (let [:a. 1
:v. (+ :a. 1)
:x. (+ :v. 2)
:z. (+ :v. :x.)]
(cond (= :v. 10) 20
(= :v. 20) 30
:else 0))}
{:k (if- (contains? :myarray 10)
(take 0 10 (conj-distinct :myarray 10))
(get :myarray 5))}
{:l (let [:a. (reduce (fn [:sum. :n.] (+ :sum. :n.))
0
(filter (fn [:w.] (some? :w.))
(map (fn [:v.] (+ :v. 1)) :myarray)))
:b. (let [:c. (into {} (ziparray (range 20) :myarray))]
(into [] (merge :c. :a.)))]
(concat :a. :b.))}])

cMQL as tool to generate MQL

This method uses the JSON portabillity,and it can be used from any driver language,
and without knowing any clojure.

Manual

Easy to use with languages that support JSON like literals like js/python etc

  • cMQL-Play write cMQL (or make a small cMQL app just to generate MQL)
  • get the generated MQL, make small changes or add driver variables
  • add the MQL in code

Auto

The above can be automated and expanded to languages like Java/Go etc that don't support JSON like literals.

  • cMQL can become MQL, driver variables can have a symbolic notation for example "$$$myvar"
  • MQL JSON can become an Document (with parse)
  • variables can be added to the Document object, from a hash-map that have {myvar : value} dynamically

The cost is practical zero(queries are so small data-structures), and it only requires simple code to auto-mate it. This method is the way to use MQL instead of driver query builders.

For more information and example on how to use cMQL as tool see also here

cMQL as API

JSON portabillity allowed us to use MQL in many drivers, but Clojure portabillity allows as to call cMQL directly.

cMQL code can be called directly from the bellow languages (without generating MQL), and its simple like a normal function call

  • Java
  • Javascript
  • Clojure
  • Clojurescript

Clojure is made to be a hosted language and using cMQL from Java or Nodejs app is like a normal function call.

This way we don't need to generate MQL, we call cMQL directly, and we can choose how much clojure/cMQL to used.
We can use cMQL only for queries or add application logic also, or even make all the application in 1 language clojure/cmql

Java programmers

Calling cMQL from Java is like a normal static java function
Getting the benefits of cMQL to write data as json literals and queries easily
cMQL is functional and suitable for data processing and querying.
cMQL can be called like a normal static java function.

Java programmers use cMQL-j.
See also Why cMQL and Why cMQL for Java?

Node programmers

Calling cMQL from js is like a normal static java function
cMQL is functional and suitable for data processing and querying.
cMQL can be called like a normal js function.

Node programmers use cMQL-js See also Why cMQL and Why cMQL for Javascript?

Clojure programmers

cMQL is like Clojure.
It feels like querying MongoDB in Clojure.
If someone knows Clojure almost knows cMQL already.

cMQL is fast because we can encode to BSON directly from Clojure,or decode directly from BSON.Perfomance is like native Java driver,even if we write or read clojure data.

Clojure programmers use cMQL-j and the Java driver.
See also Why cMQL and Why cMQL for Clojure?

ClojureScript programmers

For the same reasons as Clojure programmers, described above.

ClojureScript programmers use cMQL-js and the Nodejs driver.
See also Why cMQL and Why cMQL for Clojurescript?

cMQL-goals

  1. Be simple,readable,and much smaller
  2. Cover MQL
  3. Easy to use from drivers
  4. Run in many drivers,be portable
  5. Use the driver way
  6. Perfomance,encode/decode BSON directly to the target language data-structures
  7. Popularity
  8. Replace javascript
  9. Unify MongoDB developement in 1 language

Simple

cMQL main goal is to be

  • much smaller in code (its 2-3x for the most queries)
  • clear notation
  • clear structure

Syntax and notation

  1. {} for data (like MQL)

    {
    :firstName "Louna"
    :country "Finland"
    :city "Jyvaskyla"
    :degrees ["Nurse" "Cardio emergencies qualified"]
    }
  2. () for code (like clojure)

    (reduce (fn [sum n] (+ sum n)) [] myarray) ; clojure
    (reduce (fn [:sum. :n.] (+ :sum. :n.)) [] :myarray) ; cMQL
    {"$reduce" {"input" "$myarray", ; MQL
    "initialValue" [],
    "in" {"$let" {"vars" {"sum" "$$value", "n" "$$this"},
    "in" {"$add" ["$$sum" "$$n"]}}}}}
  3. Homoiconic
    MQL cMQL are both homoiconic,meaning that code is written in data structures of the language
    They both use a functional nested way of programming suitable for data processing

  4. Minimalistic

    • No extra words (like about "vars" "in" etc)
    • No symbols like $ $$ "" , no capital letters etc
  5. Keywords for references and variables

    :afield => "$field" reference
    :avar. => "$$avar" variable
    :avar.afield. = :.avar.afield variable path
    :REMOVE. => "$$REMOVE" system variable
    {:afield ""} => {"afield" ""} :afield on the left as key is a string
  6. Use of core language names

    We use =,> not equal gt etc ,cMQL feels like using clojure to query MongoDB
    This is made possible with clojure's powerful macros

    Inside the query we can refer to clojure with prefix like c/str.
    If we need to write alot of code,to avoid the prefix we can make it a function.

    For Clojure programmers this mean that they already know most part of cMQL
    For non Clojure programmers cMQL is easier to learn that MQL

  7. Queries/pipelines

    Queries are processed to allow the programmer to write minimal and intuitive code.
    There is no perfomance cost,because queries are so small data structures.

    • Auto add common used stages
      • filters will become match stage
      • {} is $addFields
      • [] is $project , using ! to unset and {:anewfield ...} to addfield
    • Sorting,Projecting notation :myfield :!myfield
    • nested stages becomes flatten,nil stages are removed
    (q :testdb.testcoll
    (< :salary 1000)
    (> :workingYears 1)
    {:smallChildren (filter (fn [:child.] (< :child.age. 15)) :children)}
    [:!_id :name :smallChildren.age {:new-salary (+ :salary 5)}]
    (sort-s :!new-salary))

Cover MQL

cMQL is made to cover MQL (MQL can be used raw inside cMQL also if we want)

  1. operators (see also)
  2. stages
  3. arguments
  4. commands

cMQL operators/stages/commands are part of the cMQL core and generate MQL operators/stages/commands.
cMQL arguments are used with the driver methods and are more driver depended and part of cMQL-j or cMQL-js.

Example using the Java driver and showing the alternatives The example is very small,but still code becomes 2x less.

Java

zips.aggregate(
asList(
match(eq("state", "TX")),
group("$city", sum("totalPop", "$pop")),
project(fields(excludeId(),
include("totalPop"),
computed("city", "$_id"))),
sort(descending("totalPop"))));

Clojure interop

(.aggregate zips
[(match (eq "state" "TX"))
(group "$city" (sum "totalPop" "$pop"))
(project (fields (excludeId)
(include "totalPop")
(computed "city" "$_id")))
(sort (descending "totalPop"))])

cMQL arguments (keep the driver method,but pipeline in cMQL)

(.aggregate zips
(p (= :state "TX")
(group :city {:totalPop (sum :pop))
[:!_id :totalPop]
(sort :!totalPop)))

cMQL wrap methods

(q zips
(p (= :state "TX")
(group :city {:totalPop (sum :pop))
[:!_id :totalPop]
(sort :!totalPop)))

cMQL command like call (portabillity between the drivers)

(q zips
(= :state "TX")
(group :city {:totalPop (sum :pop))
[:!_id :totalPop]
(sort :!totalPop)
{:allowDiskUse true})

cMQL is not intent to replace the driver or change the way the driver is used. cMQL is made to make queries easier.

Update and query operators

In MongoDB we have like 3 type of operators,query/update/aggregate.

This is complex because

  • they share names
  • functionality
  • they have different syntax
  • we cant mix them (exception is the $exp that allows aggregate operators in find)
    this is the main problem,for example i can try to do an update with update operator, but i am bounded to use only update operators that are so limited.

To avoid this complexity cMQL tries to use 1 way,and do the querying and updates with the aggregation framework.

  • aggregate operators inside the $match stage with $exp
    MongoDB 5 allows index use not only with $exp not only for $eq,see v5 docs
    So far to allow the use of index cMQL wraped some query operators,and if needed more can be wrapped.
  • pipeline updates (cMQL makes pipelines easy)
  • aggregation "versions" of query operators to replace them

If something cannot be done with the aggregation framework

  • use raw MQL inside cMQL
  • or use the driver with update/query operator
  • use javascript

For example we can use raw MQL query operators inside cMQL

(q :testdb.testcoll
{"$match" {"$or" [{"score" {"$gt" 70 "$lt" 90}} {"views" {"$gte" 1000}}]}}
(limit 1))

If an operator can't be replaced from the aggregation framework,and its commonly used, cMQL can change in the future to support that operator.

Easy to use from drivers

MQL is using JSON to reach developers in different languages.
cMQL is using clojure as a hosted language to reach developers in different languages.

Clojure's Interop

cMQL as a clojure library uses clojure's interop Clojure is made from the start to be a hosted language and interop with the host language is very easy

Clojure doesn't just runs on the platform,each Clojure version is made for the specific platform,for example Clojure collections are Java Collections also.

Using Java/Js from Clojure is completly easy.

For example
aggregate(acoll,apipeline);
Becomes
(.aggregate acol acoll apipeline)

The opposite calling Clojure

  • Clojurescript Clojurescript functions are javascript functions,so we simply call them
  • Clojure if the Clojure programmer made us a Java class, we can call Clojure like a Java static method if he didnt we need 1-2 lines of code to make the java wrapper,and then we call like a normal Java

cMQL uses the driver way

cMQL also "respects" the driver way, cMQL targets only the query part.
It doesnt attempt to change the way we use the driver for example how we connect to the database etc
Only exceptions are complicated things off the driver or very commonly used parts of the driver

For example cMQL doesn't support the wrap of driver methods even the query methods,but for aggregate we wrap it with the q macro,because its so commonly used.

The goal is Java/javascript programmers to continue to use the driver like they are used to do it, even if they use cMQL

cMQL is not embedded

MongoDB tries to connect MQL with programming languages with JSON as bridge,allowing MQL to be embeded in their code.
But MQL is not like Java or Javascript,its looks like clojure in JSON syntax.
And programmers soon realize that MQL is a new different language.

cMQL is not embedded,it can be used

  • same project with a folder for the clojure code (leinigein,maven etc)
  • or as a dependency,for example queries installed in maven as a jar,and we use them from Java

*exception is when the application is in Clojure/Clojurescript.

Run in many drivers

cMQL targets all projects that cMQL can support.
The vast majority of projects are in Javascript and Java and they write complex queries.

cMQL-j

  1. Java projects
    queries in cMQL,call from java
  2. Clojure projects

cMQL-js

  1. Nodejs projects
    queries in cMQL,call from js
  2. ClojureScript projects

For now only Java/Nodejs driver are supported.

cMQL is code is build to make this easier

  1. cMQL-core
    Converts cMQL to MQL
    Its driver independent,allows us to replace MQL with cMQL
    • Operators
    • Stages
    • Commands Commands can be used with runCommand of driver,or cMQL
      parts like pipelines can be used as arguments in driver methods
  2. cMQL-j
    Uses cMQL-core to convert cMQL to MQL
    Adds driver specific code for example convert a MQL pipeline to a Java Arraylist
  3. cMQL-js
    Uses cMQL-core to convert cMQL to MQL Adds driver specific code for example convert a MQL pipeline to a js array

Portability

cMQL queries are the same in all drivers.

if cMQL commands are used code is exactly the same.
if driver methods are used with cMQL arguements,code is almost exactly the same.

cMQL can act as a common query language for all the drivers it supports

Perfomance

For Java/JS the drivers provides the way to encode/decode cMQL adds a way to encode/decode for clojure/clojurescript data structures also

Encode : driver data structures -> BSON
Decode : BSON -> driver data structures

Encoding is auto detected,Clojure/Clojurescript data convert directly to BSON Decoding we set one default for our commands/methods for example to return JS data, and in each query we send we can say {:decode "cljs"} for example to return something different from the default

This allows us to have usable Clojure data structures without the perfomance penalty of BSON->JAVA->CLOJURE, cMQL goes BSON<->Clojure directly

Generating MQL from cMQL doesn't have any perfomance cost,because queries are so small data structures.

Replace Javascript

cMQL can replace Javascript,or reduce the js code size,for server side code also with the use of

  • Clojurescript (server side triggers)
  • cMQL wrappers (for js functions)
  • Clojurescript like language (for js functions)

For more see

This can unify all MongoDB development under 1 language

Unify MongoDB developement

cMQL can unify all MongoDB developement under 1 language.

  • cMQL for queries
  • Clojurescript for Javascript
  • Clojure for the application