summaryrefslogtreecommitdiffstats
path: root/node_modules/js-tokens/README.md
diff options
context:
space:
mode:
authorGravatar Piotr Russ <mail@pruss.it> 2020-11-18 23:26:45 +0100
committerGravatar Piotr Russ <mail@pruss.it> 2020-11-18 23:26:45 +0100
commit81ddf9b700bc48a1f8e472209f080f9c1d9a9b09 (patch)
tree8b959d50c5a614cbf9fcb346ed556140374d4b6d /node_modules/js-tokens/README.md
parent1870f3fdf43707a15fda0f609a021f516f45eb63 (diff)
downloadwebsite_creator-81ddf9b700bc48a1f8e472209f080f9c1d9a9b09.tar.gz
website_creator-81ddf9b700bc48a1f8e472209f080f9c1d9a9b09.tar.bz2
website_creator-81ddf9b700bc48a1f8e472209f080f9c1d9a9b09.zip
rm node_modules
Diffstat (limited to 'node_modules/js-tokens/README.md')
-rw-r--r--node_modules/js-tokens/README.md222
1 files changed, 0 insertions, 222 deletions
diff --git a/node_modules/js-tokens/README.md b/node_modules/js-tokens/README.md
deleted file mode 100644
index 5c93a88..0000000
--- a/node_modules/js-tokens/README.md
+++ /dev/null
@@ -1,222 +0,0 @@
-Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.svg?branch=master)](https://travis-ci.org/lydell/js-tokens)
-========
-
-A regex that tokenizes JavaScript.
-
-```js
-var jsTokens = require("js-tokens").default
-
-var jsString = "var foo=opts.foo;\n..."
-
-jsString.match(jsTokens)
-// ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
-```
-
-
-Installation
-============
-
-`npm install js-tokens`
-
-```js
-import jsTokens from "js-tokens"
-// or:
-var jsTokens = require("js-tokens").default
-```
-
-
-Usage
-=====
-
-### `jsTokens` ###
-
-A regex with the `g` flag that matches JavaScript tokens.
-
-The regex _always_ matches, even invalid JavaScript and the empty string.
-
-The next match is always directly after the previous.
-
-### `var token = matchToToken(match)` ###
-
-```js
-import {matchToToken} from "js-tokens"
-// or:
-var matchToToken = require("js-tokens").matchToToken
-```
-
-Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
-String, value: String}` object. The following types are available:
-
-- string
-- comment
-- regex
-- number
-- name
-- punctuator
-- whitespace
-- invalid
-
-Multi-line comments and strings also have a `closed` property indicating if the
-token was closed or not (see below).
-
-Comments and strings both come in several flavors. To distinguish them, check if
-the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
-
-Names are ECMAScript IdentifierNames, that is, including both identifiers and
-keywords. You may use [is-keyword-js] to tell them apart.
-
-Whitespace includes both line terminators and other whitespace.
-
-[is-keyword-js]: https://github.com/crissdev/is-keyword-js
-
-
-ECMAScript support
-==================
-
-The intention is to always support the latest stable ECMAScript version.
-
-If adding support for a newer version requires changes, a new version with a
-major verion bump will be released.
-
-Currently, [ECMAScript 2017] is supported.
-
-[ECMAScript 2017]: https://www.ecma-international.org/ecma-262/8.0/index.html
-
-
-Invalid code handling
-=====================
-
-Unterminated strings are still matched as strings. JavaScript strings cannot
-contain (unescaped) newlines, so unterminated strings simply end at the end of
-the line. Unterminated template strings can contain unescaped newlines, though,
-so they go on to the end of input.
-
-Unterminated multi-line comments are also still matched as comments. They
-simply go on to the end of the input.
-
-Unterminated regex literals are likely matched as division and whatever is
-inside the regex.
-
-Invalid ASCII characters have their own capturing group.
-
-Invalid non-ASCII characters are treated as names, to simplify the matching of
-names (except unicode spaces which are treated as whitespace).
-
-Regex literals may contain invalid regex syntax. They are still matched as
-regex literals. They may also contain repeated regex flags, to keep the regex
-simple.
-
-Strings may contain invalid escape sequences.
-
-
-Limitations
-===========
-
-Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
-perfect. But that’s not the point either.
-
-You may compare jsTokens with [esprima] by using `esprima-compare.js`.
-See `npm run esprima-compare`!
-
-[esprima]: http://esprima.org/
-
-### Template string interpolation ###
-
-Template strings are matched as single tokens, from the starting `` ` `` to the
-ending `` ` ``, including interpolations (whose tokens are not matched
-individually).
-
-Matching template string interpolations requires recursive balancing of `{` and
-`}`—something that JavaScript regexes cannot do. Only one level of nesting is
-supported.
-
-### Division and regex literals collision ###
-
-Consider this example:
-
-```js
-var g = 9.82
-var number = bar / 2/g
-
-var regex = / 2/g
-```
-
-A human can easily understand that in the `number` line we’re dealing with
-division, and in the `regex` line we’re dealing with a regex literal. How come?
-Because humans can look at the whole code to put the `/` characters in context.
-A JavaScript regex cannot. It only sees forwards.
-
-When the `jsTokens` regex scans throught the above, it will see the following
-at the end of both the `number` and `regex` rows:
-
-```js
-/ 2/g
-```
-
-It is then impossible to know if that is a regex literal, or part of an
-expression dealing with division.
-
-Here is a similar case:
-
-```js
-foo /= 2/g
-foo(/= 2/g)
-```
-
-The first line divides the `foo` variable with `2/g`. The second line calls the
-`foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
-sees forwards, it cannot tell the two cases apart.
-
-There are some cases where we _can_ tell division and regex literals apart,
-though.
-
-First off, we have the simple cases where there’s only one slash in the line:
-
-```js
-var foo = 2/g
-foo /= 2
-```
-
-Regex literals cannot contain newlines, so the above cases are correctly
-identified as division. Things are only problematic when there are more than
-one non-comment slash in a single line.
-
-Secondly, not every character is a valid regex flag.
-
-```js
-var number = bar / 2/e
-```
-
-The above example is also correctly identified as division, because `e` is not a
-valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
-(any letter) as flags, but it is not worth it since it increases the amount of
-ambigous cases. So only the standard `g`, `m`, `i`, `y` and `u` flags are
-allowed. This means that the above example will be identified as division as
-long as you don’t rename the `e` variable to some permutation of `gmiyu` 1 to 5
-characters long.
-
-Lastly, we can look _forward_ for information.
-
-- If the token following what looks like a regex literal is not valid after a
- regex literal, but is valid in a division expression, then the regex literal
- is treated as division instead. For example, a flagless regex cannot be
- followed by a string, number or name, but all of those three can be the
- denominator of a division.
-- Generally, if what looks like a regex literal is followed by an operator, the
- regex literal is treated as division instead. This is because regexes are
- seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
- could likely be part of such an expression.
-
-Please consult the regex source and the test cases for precise information on
-when regex or division is matched (should you need to know). In short, you
-could sum it up as:
-
-If the end of a statement looks like a regex literal (even if it isn’t), it
-will be treated as one. Otherwise it should work as expected (if you write sane
-code).
-
-
-License
-=======
-
-[MIT](LICENSE).