Stable version 0.4.6

This release is dedicated to Stardew Valley, for fixing the Harvest Moon
unplayable issue where Ann gives up on her inventions after marriage ;_;

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,27 @@
+#### For bugs
+- Rule Id (if any, e.g. SC1000): 
+- My shellcheck version (`shellcheck --version` or "online"): 
+- [ ] I tried on shellcheck.net and verified that this is still a problem on the latest commit
+- [ ] It's not reproducible on shellcheck.net, but I think that's because it's an OS, configuration or encoding issue
+
+#### For new checks and feature suggestions
+- [ ] shellcheck.net (i.e. the latest commit) currently gives no useful warnings about this
+- [ ] I searched through https://github.com/koalaman/shellcheck/issues and didn't find anything related
+
+
+#### Here's a snippet or screenshot that shows the problem:
+
+```sh
+
+#!/your/interpreter
+your script here
+
+```
+
+#### Here's what shellcheck currently says:
+
+
+
+#### Here's what I wanted or expected to see:
+
+

.travis.yml

@@ -0,0 +1,21 @@
+sudo: required
+
+language: sh
+
+services:
+  - docker
+
+before_install:
+  - export DOCKER_REPO=koalaman/shellcheck
+  - |-
+    export TAG=$([ "$TRAVIS_BRANCH" == "master" ] && echo "latest" || ([ -n "$TRAVIS_TAG" ] && echo "$TRAVIS_TAG") || echo "$TRAVIS_BRANCH")
+
+script:
+  - docker build -t builder -f Dockerfile_builder .
+  - docker run --rm -it -v $(pwd):/mnt builder
+  - docker build -t $DOCKER_REPO:$TAG .
+
+after_success:
+  - docker login -e="$DOCKER_EMAIL" -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
+  - |-
+    ([ "$TRAVIS_BRANCH" == "master" ] || [ -n "$TRAVIS_TAG" ]) && docker push "$DOCKER_REPO:$TAG"

Dockerfile

@@ -0,0 +1,11 @@
+FROM alpine:latest
+
+MAINTAINER Nikyle Nguyen <NLKNguyen@MSN.com>
+
+COPY package/bin/shellcheck /usr/local/bin/
+COPY package/lib/           /usr/local/lib/
+
+RUN ldconfig /usr/local/lib
+
+WORKDIR /mnt
+ENTRYPOINT ["shellcheck"]

Dockerfile_builder

@@ -0,0 +1,54 @@
+FROM mitchty/alpine-ghc:latest
+
+MAINTAINER Nikyle Nguyen <NLKNguyen@MSN.com>
+
+RUN apk add --no-cache build-base
+
+RUN mkdir -p /usr/src/shellcheck
+WORKDIR /usr/src/shellcheck
+
+# # ------------------------------------------------------------
+# # Build & Test
+# # ------------------------------------------------------------
+
+# Obtain the dependencies first, which are less likely to change, in order to reduce
+# subsequent build time by leveraging image cache. This benefits developers when they
+# build their code with this image locally. In case of Travis CI, this doesn't help
+# reduce building time because Travis CI doesn't use cache.
+COPY ShellCheck.cabal .
+RUN cabal update && cabal install --only-dependencies
+
+# Copy the rest of the source files, including ShellCheck.cabal again but doesn't matter
+COPY . .
+
+# Build
+RUN cabal install
+
+# Test
+RUN cabal test
+
+# # ------------------------------------------------------------
+# # Set PATH
+# # ------------------------------------------------------------
+
+# Add runtime path to easily reach the executable file. This only exists during build.
+ENV PATH "/root/.cabal/bin:$PATH"
+
+# Make it permanent for someone who login to the container of this image
+RUN echo "export PATH=${PATH}" >>  /etc/profile
+
+# # ------------------------------------------------------------
+# # Extract Binaries
+# # ------------------------------------------------------------
+
+# Get shellcheck binary
+RUN mkdir -p /package/bin/
+RUN cp $(which shellcheck) /package/bin/
+
+# Get shared libraries using magic
+RUN mkdir -p /package/lib/
+RUN ldd $(which shellcheck) | grep "=> /" | awk '{print $3}' | xargs -I '{}' cp -v '{}' /package/lib/
+
+
+# Copy shellcheck package out to mounted directory
+CMD ["cp", "-avr", "/package", "/mnt/"]

README.md

@@ -36,9 +36,9 @@ Run `shellcheck yourscript` in your terminal for instant output, as seen above.
 
 You can see ShellCheck suggestions directly in a variety of editors.
 
-* Vim, through [Syntastic](https://github.com/scrooloose/syntastic):
+* Vim, through [ALE](https://github.com/w0rp/ale) or [Syntastic](https://github.com/scrooloose/syntastic):
 
-![Screenshot of vim showing inlined shellcheck feedback](doc/vim-syntastic.png).
+![Screenshot of Vim showing inlined shellcheck feedback](doc/vim-syntastic.png).
 
 * Emacs, through [Flycheck](https://github.com/flycheck/flycheck):
 
@@ -54,8 +54,21 @@ You can see ShellCheck suggestions directly in a variety of editors.
 #### In your build or test suites
 While ShellCheck is mostly intended for interactive use, it can easily be added to builds or test suites.
 
-Use ShellCheck's exit code, or its [CheckStyle compatible XML output](shellcheck.1.md#user-content-formats). There's also a simple JSON output format for easy integration.
+ShellCheck makes canonical use of exit codes, and can output simple JSON, CheckStyle compatible XML, GCC compatible warnings as well as human readable text (with or without ANSI colors). See the [Integration](https://github.com/koalaman/shellcheck/wiki/Integration) wiki page for more documentation.
 
+## Travis CI Setup
+
+If you want to use ShellCheck in Travis CI, setting it up is simple :tada:.
+
+```yml
+language: bash
+addons:
+  apt:
+    sources:
+    - debian-sid    # Grab ShellCheck from the Debian repo
+    packages:
+    - shellcheck
+```
 
 ## Installing
 
@@ -64,12 +77,21 @@ The easiest way to install ShellCheck locally is through your package manager.
 On systems with Cabal (installs to `~/.cabal/bin`):
 
     cabal update
-    cabal install shellcheck
+    cabal install ShellCheck
 
 On Debian based distros:
 
     apt-get install shellcheck
 
+On Gentoo based distros:
+
+    emerge --ask shellcheck
+
+On EPEL based distros:
+
+    yum -y install epel-release
+    yum install ShellCheck
+
 On Fedora based distros:
 
     dnf install ShellCheck
@@ -95,6 +117,9 @@ add OBS devel:languages:haskell repository from https://build.opensuse.org/proje
 
 or use OneClickInstall - https://software.opensuse.org/package/ShellCheck
 
+From Docker Hub:
+
+    docker pull koalaman/shellcheck
 
 ## Compiling from source
 
@@ -103,7 +128,7 @@ This section describes how to build ShellCheck from a source directory. ShellChe
 
 #### Installing Cabal
 
-ShellCheck is built and packaged using Cabal. Install the package `cabal-install` from your system's package manager (with e.g. `apt-get`, `yum`, `zypper` or `brew`).
+ShellCheck is built and packaged using Cabal. Install the package `cabal-install` from your system's package manager (with e.g. `apt-get`, `brew`, `emerge`, `yum`, or `zypper`).
 
 On MacPorts, the package is instead called `hs-cabal-install`, while native Windows users should install the latest version of the Haskell platform from https://www.haskell.org/platform/
 
@@ -179,6 +204,8 @@ ShellCheck can recognize many types of incorrect test statements.
     [ $1 -eq "shellcheck" ]           # Numerical comparison of strings
     [ $n && $m ]                      # && in [ .. ]
     [ grep -q foo file ]              # Command without $(..)
+    [[ "$$file" == *.jpg ]]           # Comparisons that can't succeed
+    (( 1 -lt 2 ))                     # Using test operators in ((..))
 
 
 #### Frequently misused commands
@@ -207,9 +234,11 @@ ShellCheck recognizes many common beginner's syntax errors:
     var$n="Hello"                     # Wrong indirect assignment
     echo ${var$n}                     # Wrong indirect reference
     var=(1, 2, 3)                     # Comma separated arrays
+    array=( [index] = value )         # Incorrect index initialization
     echo "Argument 10 is $10"         # Positional parameter misreference
     if $(myfunction); then ..; fi     # Wrapping commands in $()
     else if othercondition; then ..   # Using 'else if'
+    
 
 
 #### Style
@@ -232,7 +261,8 @@ ShellCheck can recognize issues related to data and typing:
 
     args="$@"                         # Assigning arrays to strings
     files=(foo bar); echo "$files"    # Referencing arrays as strings
-    printf "%s\n" "Arguments: $@."    # Concatenating strings and arrays.
+    declare -A arr=(foo bar)          # Associative arrays without index
+    printf "%s\n" "Arguments: $@."    # Concatenating strings and arrays
     [[ $# > 2 ]]                      # Comparing numbers as strings
     var=World; echo "Hello " var      # Unused lowercase variables
     echo "Hello $name"                # Unassigned lowercase variables
@@ -265,6 +295,7 @@ ShellCheck will warn when using features not supported by the shebang. For examp
     foo-bar() { ..; }                # Undefined/unsupported function name
     [ $UID = 0 ]                     # Variable undefined in dash/sh
     local var=value                  # local is undefined in sh
+    time sleep 1 | sleep 5           # Undefined uses of 'time'
 
 
 #### Miscellaneous
@@ -275,11 +306,13 @@ ShellCheck recognizes a menagerie of other issues:
     PATH="$PATH:~/bin"                # Literal tilde in $PATH
     rm “file”                         # Unicode quotes
     echo "Hello world"                # Carriage return / DOS line endings
+    echo hello \                      # Trailing spaces after \
     var=42 echo $var                  # Expansion of inlined environment
     #!/bin/bash -x -e                 # Common shebang errors
     echo $((n/180*100))               # Unnecessary loss of precision
     ls *[:digit:].txt                 # Bad character class globs
-    sed 's/foo/bar/' file > file       # Redirecting to input
+    sed 's/foo/bar/' file > file      # Redirecting to input
+
 
 
 ## Testimonials
@@ -289,6 +322,11 @@ ShellCheck recognizes a menagerie of other issues:
 Alexander Tarasikov,
 [via Twitter](https://twitter.com/astarasikov/status/568825996532707330)
 
+## Ignoring issues
+
+Issues can be ignored via environmental variable, command line, individually or globally within a file:
+
+https://github.com/koalaman/shellcheck/wiki/Ignore
 
 ## Reporting bugs
 

ShellCheck.cabal

@@ -1,5 +1,5 @@
 Name:             ShellCheck
-Version:          0.4.4
+Version:          0.4.6
 Synopsis:         Shell script analysis tool
 License:          GPL-3
 License-file:     LICENSE
@@ -55,6 +55,7 @@ library
       ShellCheck.AnalyzerLib
       ShellCheck.Checker
       ShellCheck.Checks.Commands
+      ShellCheck.Checks.ShellSupport
       ShellCheck.Data
       ShellCheck.Formatter.Format
       ShellCheck.Formatter.CheckStyle

ShellCheck/AST.hs

@@ -21,6 +21,7 @@ module ShellCheck.AST where
 
 import Control.Monad
 import Control.Monad.Identity
+import Text.Parsec
 import qualified ShellCheck.Regex as Re
 
 data Id = Id Int deriving (Show, Eq, Ord)
@@ -32,6 +33,7 @@ data FunctionKeyword = FunctionKeyword Bool deriving (Show, Eq)
 data FunctionParentheses = FunctionParentheses Bool deriving (Show, Eq)
 data CaseType = CaseBreak | CaseFallThrough | CaseContinue deriving (Show, Eq)
 
+data Root = Root Token
 data Token =
     TA_Binary Id String Token Token
     | TA_Assignment Id String Token Token
@@ -43,15 +45,17 @@ data Token =
     | TC_And Id ConditionType String Token Token
     | TC_Binary Id ConditionType String Token Token
     | TC_Group Id ConditionType Token
-    | TC_Noary Id ConditionType Token
+    | TC_Nullary Id ConditionType Token
     | TC_Or Id ConditionType String Token Token
     | TC_Unary Id ConditionType String Token
     | T_AND_IF Id
     | T_AndIf Id (Token) (Token)
     | T_Arithmetic Id Token
     | T_Array Id [Token]
-    | T_IndexedElement Id Token Token
-    | T_Assignment Id AssignmentMode String (Maybe Token) Token
+    | T_IndexedElement Id [Token] Token
+    -- Store the index as string, and parse as arithmetic or string later
+    | T_UnparsedIndex Id SourcePos String
+    | T_Assignment Id AssignmentMode String [Token] Token
     | T_Backgrounded Id Token
     | T_Backticked Id [Token]
     | T_Bang Id
@@ -96,6 +100,7 @@ data Token =
     | T_IfExpression Id [([Token],[Token])] [Token]
     | T_In  Id
     | T_IoFile Id Token Token
+    | T_IoDuplicate Id Token String
     | T_LESSAND Id
     | T_LESSGREAT Id
     | T_Lbrace Id
@@ -106,6 +111,7 @@ data Token =
     | T_NormalWord Id [Token]
     | T_OR_IF Id
     | T_OrIf Id (Token) (Token)
+    | T_ParamSubSpecialChar Id String -- e.g. '%' in ${foo%bar}  or '/' in ${foo/bar/baz}
     | T_Pipeline Id [Token] [Token] -- [Pipe separators] [Commands]
     | T_ProcSub Id String [Token]
     | T_Rbrace Id
@@ -145,7 +151,7 @@ tokenEquals a b = kludge a == kludge b
 instance Eq Token where
     (==) = tokenEquals
 
-analyze :: Monad m => (Token -> m ()) -> (Token -> m ()) -> (Token -> Token) -> Token -> m Token
+analyze :: Monad m => (Token -> m ()) -> (Token -> m ()) -> (Token -> m Token) -> Token -> m Token
 analyze f g i =
     round
   where
@@ -153,7 +159,7 @@ analyze f g i =
         f t
         newT <- delve t
         g t
-        return . i $ newT
+        i newT
     roundAll = mapM round
 
     roundMaybe Nothing = return Nothing
@@ -186,14 +192,18 @@ analyze f g i =
     delve (T_DollarArithmetic id c) = d1 c $ T_DollarArithmetic id
     delve (T_DollarBracket id c) = d1 c $ T_DollarBracket id
     delve (T_IoFile id op file) = d2 op file $ T_IoFile id
+    delve (T_IoDuplicate id op num) = d1 op $ \x -> T_IoDuplicate id x num
     delve (T_HereString id word) = d1 word $ T_HereString id
     delve (T_FdRedirect id v t) = d1 t $ T_FdRedirect id v
-    delve (T_Assignment id mode var index value) = do
-        a <- roundMaybe index
+    delve (T_Assignment id mode var indices value) = do
+        a <- roundAll indices
         b <- round value
         return $ T_Assignment id mode var a b
     delve (T_Array id t) = dl t $ T_Array id
-    delve (T_IndexedElement id t1 t2) = d2 t1 t2 $ T_IndexedElement id
+    delve (T_IndexedElement id indices t) = do
+        a <- roundAll indices
+        b <- round t
+        return $ T_IndexedElement id a b
     delve (T_Redirecting id redirs cmd) = do
         a <- roundAll redirs
         b <- round cmd
@@ -248,7 +258,7 @@ analyze f g i =
     delve (TC_Group id typ token) = d1 token $ TC_Group id typ
     delve (TC_Binary id typ op lhs rhs) = d2 lhs rhs $ TC_Binary id typ op
     delve (TC_Unary id typ op token) = d1 token $ TC_Unary id typ op
-    delve (TC_Noary id typ token) = d1 token $ TC_Noary id typ
+    delve (TC_Nullary id typ token) = d1 token $ TC_Nullary id typ
 
     delve (TA_Binary id op t1 t2) = d2 t1 t2 $ TA_Binary id op
     delve (TA_Assignment id op t1 t2) = d2 t1 t2 $ TA_Assignment id op
@@ -310,8 +320,10 @@ getId t = case t of
         T_DollarBraced id _  -> id
         T_DollarArithmetic id _  -> id
         T_BraceExpansion id _  -> id
+        T_ParamSubSpecialChar id _ -> id
         T_DollarBraceCommandExpansion id _  -> id
         T_IoFile id _ _  -> id
+        T_IoDuplicate id _ _  -> id
         T_HereDoc id _ _ _ _ -> id
         T_HereString id _  -> id
         T_FdRedirect id _ _  -> id
@@ -344,7 +356,7 @@ getId t = case t of
         TC_Group id _ _  -> id
         TC_Binary id _ _ _ _  -> id
         TC_Unary id _ _ _  -> id
-        TC_Noary id _ _  -> id
+        TC_Nullary id _ _  -> id
         TA_Binary id _ _ _  -> id
         TA_Assignment id _ _ _  -> id
         TA_Unary id _ _  -> id
@@ -363,10 +375,11 @@ getId t = case t of
         T_CoProc id _ _ -> id
         T_CoProcBody id _ -> id
         T_Include id _ _ -> id
+        T_UnparsedIndex id _ _ -> id
 
 blank :: Monad m => Token -> m ()
 blank = const $ return ()
-doAnalysis f = analyze f blank id
-doStackAnalysis startToken endToken = analyze startToken endToken id
-doTransform i = runIdentity . analyze blank blank i
+doAnalysis f = analyze f blank return
+doStackAnalysis startToken endToken = analyze startToken endToken return
+doTransform i = runIdentity . analyze blank blank (return . i)
 

ShellCheck/ASTLib.hs

@@ -21,7 +21,9 @@ module ShellCheck.ASTLib where
 
 import ShellCheck.AST
 
+import Control.Monad.Writer
 import Control.Monad
+import Data.Functor
 import Data.List
 import Data.Maybe
 
@@ -54,6 +56,8 @@ isGlob _ = False
 -- Is this shell word a constant?
 isConstant token =
     case token of
+        -- This ignores some cases like ~"foo":
+        T_NormalWord _ (T_Literal _ ('~':_) : _)  -> False
         T_NormalWord _ l   -> all isConstant l
         T_DoubleQuoted _ l -> all isConstant l
         T_SingleQuoted _ _ -> True
@@ -83,13 +87,14 @@ oversimplify token =
         (T_Glob _ s) -> [s]
         (T_Pipeline _ _ [x]) -> oversimplify x
         (T_Literal _ x) -> [x]
+        (T_ParamSubSpecialChar _ x) -> [x]
         (T_SimpleCommand _ vars words) -> concatMap oversimplify words
         (T_Redirecting _ _ foo) -> oversimplify foo
         (T_DollarSingleQuoted _ s) -> [s]
         (T_Annotation _ _ s) -> oversimplify s
         -- Workaround for let "foo = bar" parsing
         (TA_Sequence _ [TA_Expansion _ v]) -> concatMap oversimplify v
-        otherwise -> []
+        _ -> []
 
 
 -- Turn a SimpleCommand foo -avz --bar=baz into args "a", "v", "z", "bar",
@@ -108,8 +113,11 @@ getFlagsUntil _ _ = error "Internal shellcheck error, please report! (getFlags o
 
 -- Get all flags in a GNU way, up until --
 getAllFlags = getFlagsUntil (== "--")
--- Get all flags in a BSD way, up until first non-flag argument
-getLeadingFlags = getFlagsUntil (not . ("-" `isPrefixOf`))
+-- Get all flags in a BSD way, up until first non-flag argument or --
+getLeadingFlags = getFlagsUntil (\x -> x == "--" || (not $ "-" `isPrefixOf` x))
+
+-- Check if a command has a flag.
+hasFlag cmd str = str `elem` (map snd $ getAllFlags cmd)
 
 
 -- Given a T_DollarBraced, return a simplified version of the string contents.
@@ -167,6 +175,20 @@ getUnquotedLiteral (T_NormalWord _ list) =
     str _ = Nothing
 getUnquotedLiteral _ = Nothing
 
+-- Get the last unquoted T_Literal in a word like "${var}foo"THIS
+-- or nothing if the word does not end in an unquoted literal.
+getTrailingUnquotedLiteral :: Token -> Maybe Token
+getTrailingUnquotedLiteral t =
+    case t of
+        (T_NormalWord _ list@(_:_)) ->
+            from (last list)
+        _ -> Nothing
+  where
+    from t =
+        case t of
+            (T_Literal {}) -> return t
+            _ -> Nothing
+
 -- Maybe get the literal string of this token and any globs in it.
 getGlobOrLiteralString = getLiteralStringExt f
   where
@@ -185,6 +207,7 @@ getLiteralStringExt more = g
     g (TA_Expansion _ l) = allInList l
     g (T_SingleQuoted _ s) = return s
     g (T_Literal _ s) = return s
+    g (T_ParamSubSpecialChar _ s) = return s
     g x = more x
 
 -- Is this token a string literal?
@@ -194,6 +217,8 @@ isLiteral t = isJust $ getLiteralString t
 -- Turn a NormalWord like foo="bar $baz" into a series of constituent elements like [foo=,bar ,$baz]
 getWordParts (T_NormalWord _ l)   = concatMap getWordParts l
 getWordParts (T_DoubleQuoted _ l) = l
+-- TA_Expansion is basically T_NormalWord for arithmetic expressions
+getWordParts (TA_Expansion _ l)   = concatMap getWordParts l
 getWordParts other                = [other]
 
 -- Return a list of NormalWords that would result from brace expansion
@@ -206,14 +231,39 @@ braceExpand (T_NormalWord id list) = take 1000 $ do
         braceExpand item
     part x = return x
 
--- Maybe get the command name of a token representing a command
-getCommandName t =
+-- Maybe get a SimpleCommand from immediate wrappers like T_Redirections
+getCommand t =
     case t of
-        T_Redirecting _ _ w -> getCommandName w
-        T_SimpleCommand _ _ (w:_) -> getLiteralString w
-        T_Annotation _ _ t -> getCommandName t
+        T_Redirecting _ _ w -> getCommand w
+        T_SimpleCommand _ _ (w:_) -> return t
+        T_Annotation _ _ t -> getCommand t
         otherwise -> Nothing
 
+-- Maybe get the command name of a token representing a command
+getCommandName t = do
+    (T_SimpleCommand _ _ (w:rest)) <- getCommand t
+    s <- getLiteralString w
+    if "busybox" `isSuffixOf` s
+        then
+            case rest of
+                (applet:_) -> getLiteralString applet
+                _ -> return s
+        else
+            return s
+
+-- If a command substitution is a single command, get its name.
+--  $(date +%s) = Just "date"
+getCommandNameFromExpansion :: Token -> Maybe String
+getCommandNameFromExpansion t =
+    case t of
+        T_DollarExpansion _ [c] -> extract c
+        T_Backticked _ [c] -> extract c
+        T_DollarBraceCommandExpansion _ [c] -> extract c
+        otherwise -> Nothing
+  where
+    extract (T_Pipeline _ _ [cmd]) = getCommandName cmd
+    extract _ = Nothing
+
 -- Get the basename of a token representing a command
 getCommandBasename = liftM basename . getCommandName
   where
@@ -237,8 +287,10 @@ isOnlyRedirection t =
 
 isFunction t = case t of T_Function {} -> True; _ -> False
 
--- Get the list of commands from tokens that contain them, such as
--- the body of while loops and if statements.
+isBraceExpansion t = case t of T_BraceExpansion {} -> True; _ -> False
+
+-- Get the lists of commands from tokens that contain them, such as
+-- the body of while loops or branches of if statements.
 getCommandSequences t =
     case t of
         T_Script _ _ cmds -> [cmds]
@@ -251,3 +303,84 @@ getCommandSequences t =
         T_IfExpression _ thens elses -> map snd thens ++ [elses]
         otherwise -> []
 
+-- Get a list of names of associative arrays
+getAssociativeArrays t =
+    nub . execWriter $ doAnalysis f t
+  where
+    f :: Token -> Writer [String] ()
+    f t@(T_SimpleCommand {}) = fromMaybe (return ()) $ do
+        name <- getCommandName t
+        guard $ name == "declare" || name == "typeset"
+        let flags = getAllFlags t
+        guard $ elem "A" $ map snd flags
+        let args = map fst . filter ((==) "" . snd) $ flags
+        let names = mapMaybe (getLiteralStringExt nameAssignments) args
+        return $ tell names
+    f _ = return ()
+
+    nameAssignments t =
+        case t of
+            T_Assignment _ _ name _ _ -> return name
+            otherwise -> Nothing
+
+-- A Pseudoglob is a wildcard pattern used for checking if a match can succeed.
+-- For example, [[ $(cmd).jpg == [a-z] ]] will give the patterns *.jpg and ?, which
+-- can be proven never to match.
+data PseudoGlob = PGAny | PGMany | PGChar Char
+    deriving (Eq, Show)
+
+-- Turn a word into a PG pattern, replacing all unknown/runtime values with
+-- PGMany.
+wordToPseudoGlob :: Token -> Maybe [PseudoGlob]
+wordToPseudoGlob word =
+    simplifyPseudoGlob <$> concat <$> mapM f (getWordParts word)
+  where
+    f x = case x of
+        T_Literal _ s -> return $ map PGChar s
+        T_SingleQuoted _ s -> return $ map PGChar s
+
+        T_DollarBraced {} -> return [PGMany]
+        T_DollarExpansion {} -> return [PGMany]
+        T_Backticked {} -> return [PGMany]
+
+        T_Glob _ "?" -> return [PGAny]
+        T_Glob _ ('[':_)  -> return [PGAny]
+        T_Glob {} -> return [PGMany]
+
+        T_Extglob {} -> return [PGMany]
+
+        _ -> return [PGMany]
+
+-- Reorder a PseudoGlob for more efficient matching, e.g.
+-- f?*?**g -> f??*g
+simplifyPseudoGlob :: [PseudoGlob] -> [PseudoGlob]
+simplifyPseudoGlob = f
+  where
+    f [] = []
+    f (x@(PGChar _) : rest ) = x : f rest
+    f list =
+        let (anys, rest) = span (\x -> x == PGMany || x == PGAny) list in
+            order anys ++ f rest
+
+    order s = let (any, many) = partition (== PGAny) s in
+        any ++ take 1 many
+
+-- Check whether the two patterns can ever overlap.
+pseudoGlobsCanOverlap :: [PseudoGlob] -> [PseudoGlob] -> Bool
+pseudoGlobsCanOverlap = matchable
+  where
+    matchable x@(xf:xs) y@(yf:ys) =
+        case (xf, yf) of
+            (PGMany, _) -> matchable x ys || matchable xs y
+            (_, PGMany) -> matchable x ys || matchable xs y
+            (PGAny, _) -> matchable xs ys
+            (_, PGAny) -> matchable xs ys
+            (_, _) -> xf == yf && matchable xs ys
+
+    matchable [] [] = True
+    matchable (PGMany : rest) [] = matchable rest []
+    matchable (_:_) [] = False
+    matchable [] r = matchable r []
+
+wordsCanBeEqual x y = fromMaybe True $
+    liftM2 pseudoGlobsCanOverlap (wordToPseudoGlob x) (wordToPseudoGlob y)

ShellCheck/Analyzer.hs

@@ -23,7 +23,9 @@ import ShellCheck.Analytics
 import ShellCheck.AnalyzerLib
 import ShellCheck.Interface
 import Data.List
+import Data.Monoid
 import qualified ShellCheck.Checks.Commands
+import qualified ShellCheck.Checks.ShellSupport
 
 
 -- TODO: Clean up the cruft this is layered on
@@ -32,5 +34,12 @@ analyzeScript spec = AnalysisResult {
     arComments =
         filterByAnnotation (asScript spec) . nub $
             runAnalytics spec
-            ++ ShellCheck.Checks.Commands.runChecks spec
+            ++ runChecker params (checkers params)
 }
+  where
+    params = makeParameters spec
+
+checkers params = mconcat $ map ($ params) [
+    ShellCheck.Checks.Commands.checker,
+    ShellCheck.Checks.ShellSupport.checker
+    ]

ShellCheck/AnalyzerLib.hs

@@ -29,7 +29,7 @@ import ShellCheck.Regex
 
 import Control.Arrow (first)
 import Control.Monad.Identity
-import Control.Monad.Reader
+import Control.Monad.RWS
 import Control.Monad.State
 import Control.Monad.Writer
 import Data.Char
@@ -40,16 +40,48 @@ import qualified Data.Map as Map
 import Test.QuickCheck.All (forAllProperties)
 import Test.QuickCheck.Test (quickCheckWithResult, stdArgs, maxSuccess)
 
-type Analysis = ReaderT Parameters (Writer [TokenComment]) ()
+type Analysis = AnalyzerM ()
+type AnalyzerM a = RWS Parameters [TokenComment] Cache a
+nullCheck = const $ return ()
 
 
+data Checker = Checker {
+    perScript :: Root -> Analysis,
+    perToken :: Token -> Analysis
+}
+
+runChecker :: Parameters -> Checker -> [TokenComment]
+runChecker params checker = notes
+    where
+        root = rootNode params
+        check = perScript checker `composeAnalyzers` (\(Root x) -> void $ doAnalysis (perToken checker) x)
+        notes = snd $ evalRWS (check $ Root root) params Cache
+
+instance Monoid Checker where
+    mempty = Checker {
+        perScript = nullCheck,
+        perToken = nullCheck
+        }
+    mappend x y = Checker {
+        perScript = perScript x `composeAnalyzers` perScript y,
+        perToken = perToken x `composeAnalyzers` perToken y
+        }
+
+
+composeAnalyzers :: (a -> Analysis) -> (a -> Analysis) -> a -> Analysis
+composeAnalyzers f g x = f x >> g x
+
 data Parameters = Parameters {
     variableFlow :: [StackData],
     parentMap :: Map.Map Id Token,
     shellType :: Shell,
-    shellTypeSpecified :: Bool
+    shellTypeSpecified :: Bool,
+    rootNode :: Token
     }
 
+-- TODO: Cache results of common AST ops here
+data Cache = Cache {}
+
 data Scope = SubshellScope String | NoneScope deriving (Show, Eq)
 data StackData =
     StackScope Scope
@@ -81,6 +113,14 @@ pScript s =
     }
   in prRoot . runIdentity $ parseScript (mockedSystemInterface []) pSpec
 
+-- For testing. If parsed, returns whether there are any comments
+producesComments :: Checker -> String -> Maybe Bool
+producesComments c s = do
+        root <- pScript s
+        let spec = defaultSpec root
+        let params = makeParameters spec
+        return . not . null $ runChecker params c
+
 makeComment :: Severity -> Id -> Code -> String -> TokenComment
 makeComment severity id code note =
     TokenComment id $ Comment severity code note
@@ -95,6 +135,7 @@ style id code str = addComment $ makeComment StyleC id code str
 
 makeParameters spec =
     let params = Parameters {
+        rootNode = root,
         shellType = fromMaybe (determineShell root) $ asShellType spec,
         shellTypeSpecified = isJust $ asShellType spec,
         parentMap = getParentTree root,
@@ -112,6 +153,7 @@ prop_determineShell4 = determineShell (fromJust $ pScript
 prop_determineShell5 = determineShell (fromJust $ pScript
     "#shellcheck shell=sh\nfoo") == Sh
 prop_determineShell6 = determineShell (fromJust $ pScript "#! /bin/sh") == Sh
+prop_determineShell7 = determineShell (fromJust $ pScript "#! /bin/ash") == Dash
 determineShell t = fromMaybe Bash $ do
     shellString <- foldl mplus Nothing $ getCandidates t
     shellForExecutable shellString
@@ -125,8 +167,13 @@ determineShell t = fromMaybe Bash $ do
     getCandidates (T_Annotation _ annotations s) =
         map forAnnotation annotations ++
            [Just $ fromShebang s]
-    fromShebang (T_Script _ s t) = shellFor s
+    fromShebang (T_Script _ s t) = executableFromShebang s
 
+-- Given a string like "/bin/bash" or "/usr/bin/env dash",
+-- return the shell basename like "bash" or "dash"
+executableFromShebang :: String -> String
+executableFromShebang = shellFor
+  where
     shellFor s | "/env " `isInfixOf` s = head (drop 1 (words s)++[""])
     shellFor s | ' ' `elem` s = shellFor $ takeWhile (/= ' ') s
     shellFor s = reverse . takeWhile (/= '/') . reverse $ s
@@ -170,16 +217,13 @@ isQuoteFreeNode strict tree t =
     -- Are any subnodes inherently self-quoting?
     isQuoteFreeContext t =
         case t of
-            TC_Noary _ DoubleBracket _ -> return True
+            TC_Nullary _ DoubleBracket _ -> return True
             TC_Unary _ DoubleBracket _ _ -> return True
             TC_Binary _ DoubleBracket _ _ _ -> return True
             TA_Sequence {} -> return True
             T_Arithmetic {} -> return True
             T_Assignment {} -> return True
-            T_Redirecting {} -> return $
-                if strict then False else
-                    -- Not true, just a hack to prevent warning about non-expansion refs
-                    any (isCommand t) ["local", "declare", "typeset", "export", "trap", "readonly"]
+            T_Redirecting {} -> return False
             T_DoubleQuoted _ _ -> return True
             T_DollarDoubleQuoted _ _ -> return True
             T_CaseExpression {} -> return True
@@ -211,6 +255,10 @@ getClosestCommand tree t =
     getCommand t@(T_Redirecting {}) = return t
     getCommand _ = Nothing
 
+getClosestCommandM t = do
+    tree <- asks parentMap
+    return $ getClosestCommand tree t
+
 usedAsCommandName tree token = go (getId token) (tail $ getPath tree token)
   where
     go currentId (T_NormalWord id [word]:rest)
@@ -227,6 +275,12 @@ getPath tree t = t :
         Nothing -> []
         Just parent -> getPath tree parent
 
+-- Version of the above taking the map from the current context
+-- Todo: give this the name "getPath"
+getPathM t = do
+    map <- asks parentMap
+    return $ getPath map t
+
 isParentOf tree parent child =
     elem (getId parent) . map getId $ getPath tree child
 
@@ -334,6 +388,12 @@ getModifiedVariables t =
             name <- getLiteralString lhs
             return (t, t, name, DataString $ SourceFrom [rhs])
 
+        T_DollarBraced _ l -> maybeToList $ do
+            let string = bracedString t
+            let modifier = getBracedModifier string
+            guard $ ":=" `isPrefixOf` modifier
+            return (t, t, getBracedReference string, DataString $ SourceFrom [l])
+
         t@(T_FdRedirect _ ('{':var) op) -> -- {foo}>&2 modifies foo
             [(t, t, takeWhile (/= '}') var, DataString SourceInteger) | not $ isClosingFileOp op]
 
@@ -341,6 +401,7 @@ getModifiedVariables t =
             [(t, t, fromMaybe "COPROC" name, DataArray SourceInteger)]
 
         --Points to 'for' rather than variable
+        T_ForIn id str [] _ -> [(t, t, str, DataString $ SourceExternal)]
         T_ForIn id str words _ -> [(t, t, str, DataString $ SourceFrom words)]
         T_SelectIn id str words _ -> [(t, t, str, DataString $ SourceFrom words)]
         _ -> []
@@ -361,7 +422,10 @@ getReferencedVariableCommand base@(T_SimpleCommand _ _ (T_NormalWord _ (T_Litera
         "declare" -> if any (`elem` flags) ["x", "p"]
             then concatMap getReference rest
             else []
-        "readonly" -> concatMap getReference rest
+        "readonly" ->
+            if any (`elem` flags) ["f", "p"]
+            then []
+            else concatMap getReference rest
         "trap" ->
             case rest of
                 head:_ -> map (\x -> (head, head, x)) $ getVariablesFromLiteralToken head
@@ -395,7 +459,10 @@ getModifiedVariableCommand base@(T_SimpleCommand _ _ (T_NormalWord _ (T_Literal
         "typeset" -> declaredVars
 
         "local" -> concatMap getModifierParamString rest
-        "readonly" -> concatMap getModifierParamString rest
+        "readonly" ->
+            if any (`elem` flags) ["f", "p"]
+            then []
+            else concatMap getModifierParamString rest
         "set" -> maybeToList $ do
             params <- getSetParams rest
             return (base, base, "@", DataString $ SourceFrom params)
@@ -474,11 +541,20 @@ getIndexReferences s = fromMaybe [] $ do
   where
     re = mkRegex "(\\[.*\\])"
 
+getOffsetReferences mods = fromMaybe [] $ do
+    match <- matchRegex re mods
+    offsets <- match !!! 0
+    return $ matchAllStrings variableNameRegex offsets
+  where
+    re = mkRegex "^ *:(.*)"
+
 getReferencedVariables parents t =
     case t of
         T_DollarBraced id l -> let str = bracedString t in
             (t, t, getBracedReference str) :
-                map (\x -> (l, l, x)) (getIndexReferences str)
+                map (\x -> (l, l, x)) (
+                    getIndexReferences str
+                    ++ getOffsetReferences (getBracedModifier str))
         TA_Expansion id _ ->
             if isArithmeticAssignment t
             then []
@@ -520,7 +596,7 @@ getReferencedVariables parents t =
     isDereferencing = (`elem` ["-eq", "-ne", "-lt", "-le", "-gt", "-ge"])
 
     isArithmeticAssignment t = case getPath parents t of
-        this: TA_Assignment _ "=" _ _ :_ -> True
+        this: TA_Assignment _ "=" lhs _ :_ -> lhs == t
         _ -> False
 
 dataTypeFrom defaultType v = (case v of T_Array {} -> DataArray; _ -> defaultType) $ SourceFrom [v]
@@ -573,6 +649,7 @@ prop_getBracedReference9 = getBracedReference "foo:-bar" == "foo"
 prop_getBracedReference10= getBracedReference "foo: -1" == "foo"
 prop_getBracedReference11= getBracedReference "!os*" == ""
 prop_getBracedReference12= getBracedReference "!os?bar**" == ""
+prop_getBracedReference13= getBracedReference "foo[bar]" == "foo"
 getBracedReference s = fromMaybe s $
     nameExpansion s `mplus` takeName noPrefix `mplus` getSpecial noPrefix `mplus` getSpecial s
   where
@@ -595,6 +672,20 @@ getBracedReference s = fromMaybe s $
         return ""
     nameExpansion _ = Nothing
 
+prop_getBracedModifier1 = getBracedModifier "foo:bar:baz" == ":bar:baz"
+prop_getBracedModifier2 = getBracedModifier "!var:-foo" == ":-foo"
+prop_getBracedModifier3 = getBracedModifier "foo[bar]" == "[bar]"
+getBracedModifier s = fromMaybe "" . listToMaybe $ do
+    let var = getBracedReference s
+    a <- dropModifier s
+    dropPrefix var a
+  where
+    dropPrefix [] t = return t
+    dropPrefix (a:b) (c:d) | a == c = dropPrefix b d
+    dropPrefix _ _ = []
+
+    dropModifier (c:rest) | c `elem` "#!" = [rest, c:rest]
+    dropModifier x = [x]
 
 -- Useful generic functions
 potentially :: Monad m => Maybe (m ()) -> m ()
@@ -608,6 +699,10 @@ headOrDefault def _ = def
         [] -> Nothing
         (r:_) -> Just r
 
+-- Run a command if the shell is in the given list
+whenShell l c = do
+    shell <- asks shellType
+    when (shell `elem` l ) c
 
 
 filterByAnnotation token =
@@ -628,5 +723,5 @@ filterByAnnotation token =
     getCode (TokenComment _ (Comment _ c _)) = c
 
 
-return [] 
+return []
 runTests =  $( [| $(forAllProperties) (quickCheckWithResult (stdArgs { maxSuccess = 1 }) ) |])

ShellCheck/Checker.hs

@@ -39,7 +39,7 @@ import Test.QuickCheck.All
 
 tokenToPosition map (TokenComment id c) = fromMaybe fail $ do
     position <- Map.lookup id map
-    return $ PositionedComment position c
+    return $ PositionedComment position position c
   where
     fail = error "Internal shellcheck error: id doesn't exist. Please report!"
 
@@ -65,13 +65,13 @@ checkScript sys spec = do
         return . nub . sortMessages . filter shouldInclude $
             (parseMessages ++ map translator analysisMessages)
 
-    shouldInclude (PositionedComment _ (Comment _ code _)) =
+    shouldInclude (PositionedComment _ _ (Comment _ code _)) =
         code `notElem` csExcludedWarnings spec
 
     sortMessages = sortBy (comparing order)
-    order (PositionedComment pos (Comment severity code message)) =
+    order (PositionedComment pos _ (Comment severity code message)) =
         (posFile pos, posLine pos, posColumn pos, severity, code, message)
-    getPosition (PositionedComment pos _) = pos
+    getPosition (PositionedComment pos _ _) = pos
 
     analysisSpec root =
         AnalysisSpec {
@@ -84,7 +84,7 @@ getErrors sys spec =
     sort . map getCode . crComments $
         runIdentity (checkScript sys spec)
   where
-    getCode (PositionedComment _ (Comment _ code _)) = code
+    getCode (PositionedComment _ _ (Comment _ code _)) = code
 
 check = checkWithIncludes []
 
@@ -158,5 +158,25 @@ prop_sourceDirectiveDoesntFollowFile =
                 [("foo", "source bar"), ("bar", "baz=3")]
                 "#shellcheck source=foo\n. \"$1\"; echo \"$baz\""
 
+prop_filewideAnnotationBase = [2086] == check "#!/bin/sh\necho $1"
+prop_filewideAnnotation1 = null $
+    check "#!/bin/sh\n# shellcheck disable=2086\necho $1"
+prop_filewideAnnotation2 = null $
+    check "#!/bin/sh\n# shellcheck disable=2086\ntrue\necho $1"
+prop_filewideAnnotation3 = null $
+    check "#!/bin/sh\n#unerlated\n# shellcheck disable=2086\ntrue\necho $1"
+prop_filewideAnnotation4 = null $
+    check "#!/bin/sh\n# shellcheck disable=2086\n#unrelated\ntrue\necho $1"
+prop_filewideAnnotation5 = null $
+    check "#!/bin/sh\n\n\n\n#shellcheck disable=2086\ntrue\necho $1"
+prop_filewideAnnotation6 = null $
+    check "#shellcheck shell=sh\n#unrelated\n#shellcheck disable=2086\ntrue\necho $1"
+prop_filewideAnnotation7 = null $
+    check "#!/bin/sh\n# shellcheck disable=2086\n#unrelated\ntrue\necho $1"
+
+prop_filewideAnnotationBase2 = [2086, 2181] == check "true\n[ $? == 0 ] && echo $1"
+prop_filewideAnnotation8 = null $
+    check "# Disable $? warning\n#shellcheck disable=SC2181\n# Disable quoting warning\n#shellcheck disable=2086\ntrue\n[ $? == 0 ] && echo $1"
+
 return []
 runTests = $quickCheckAll

ShellCheck/Checks/Commands.hs

@@ -21,7 +21,7 @@
 {-# LANGUAGE FlexibleContexts #-}
 
 -- This module contains checks that examine specific commands by name.
-module ShellCheck.Checks.Commands (runChecks
+module ShellCheck.Checks.Commands (checker
     , ShellCheck.Checks.Commands.runTests
 ) where
 
@@ -34,8 +34,7 @@ import ShellCheck.Parser
 import ShellCheck.Regex
 
 import Control.Monad
-import Control.Monad.Reader
-import Control.Monad.Writer
+import Control.Monad.RWS
 import Data.Char
 import Data.List
 import Data.Maybe
@@ -49,22 +48,10 @@ data CommandName = Exactly String | Basename String
 data CommandCheck =
     CommandCheck CommandName (Token -> Analysis)
 
-nullCheck :: Token -> Analysis
-nullCheck _ = return ()
-
 
 verify :: CommandCheck -> String -> Bool
-verify f s = producesComments f s == Just True
-verifyNot f s = producesComments f s == Just False
-
-producesComments :: CommandCheck -> String -> Maybe Bool
-producesComments f s = do
-        root <- pScript s
-        return . not . null $ runList (defaultSpec root) [f]
-
-composeChecks f g t = do
-    f t
-    g t
+verify f s = producesComments (getChecker [f]) s == Just True
+verifyNot f s = producesComments (getChecker [f]) s == Just False
 
 arguments (T_SimpleCommand _ _ (cmd:args)) = args
 
@@ -90,13 +77,21 @@ commandChecks = [
     ,checkExportedExpansions
     ,checkAliasesUsesArgs
     ,checkAliasesExpandEarly
+    ,checkUnsetGlobs
+    ,checkFindWithoutPath
+    ,checkTimeParameters
+    ,checkTimedCommand
+    ,checkLocalScope
+    ,checkDeprecatedTempfile
+    ,checkDeprecatedEgrep
+    ,checkDeprecatedFgrep
     ]
 
 buildCommandMap :: [CommandCheck] -> Map.Map CommandName (Token -> Analysis)
 buildCommandMap = foldl' addCheck Map.empty
   where
     addCheck map (CommandCheck name function) =
-        Map.insertWith' composeChecks name function map
+        Map.insertWith' composeAnalyzers name function map
 
 
 checkCommand :: Map.Map CommandName (Token -> Analysis) -> Token -> Analysis
@@ -114,15 +109,17 @@ checkCommand map t@(T_SimpleCommand id _ (cmd:rest)) = fromMaybe (return ()) $ d
     basename = reverse . takeWhile (/= '/') . reverse
 checkCommand _ _ = return ()
 
-runList spec list = notes
-    where
-        root = asScript spec
-        params = makeParameters spec
-        notes = execWriter $ runReaderT (doAnalysis (checkCommand map) root) params
-        map = buildCommandMap list
+getChecker :: [CommandCheck] -> Checker
+getChecker list = Checker {
+    perScript = const $ return (),
+    perToken = checkCommand map
+    }
+  where
+    map = buildCommandMap list
 
-runChecks spec = runList spec commandChecks
 
+checker :: Parameters -> Checker
+checker params = getChecker commandChecks
 
 prop_checkTr1 = verify checkTr "tr [a-f] [A-F]"
 prop_checkTr2 = verify checkTr "tr 'a-z' 'A-Z'"
@@ -150,7 +147,7 @@ checkTr = CommandCheck (Basename "tr") (mapM_ f . arguments)
             info (getId word) 2020 "tr replaces sets of chars, not words (mentioned due to duplicates)."
           unless ("[:" `isPrefixOf` s) $
             when ("[" `isPrefixOf` s && "]" `isSuffixOf` s && (length s > 2) && ('*' `notElem` s)) $
-              info (getId word) 2021 "Don't use [] around ranges in tr, it replaces literal square brackets."
+              info (getId word) 2021 "Don't use [] around classes in tr, it replaces literal square brackets."
         Nothing -> return ()
 
     duplicated s =
@@ -197,23 +194,27 @@ prop_checkGrepRe8 = verify checkGrepRe "ls | grep foo*.jpg"
 prop_checkGrepRe9 = verifyNot checkGrepRe "grep '[0-9]*' file"
 prop_checkGrepRe10= verifyNot checkGrepRe "grep '^aa*' file"
 prop_checkGrepRe11= verifyNot checkGrepRe "grep --include=*.png foo"
+prop_checkGrepRe12= verifyNot checkGrepRe "grep -F 'Foo*' file"
 
-checkGrepRe = CommandCheck (Basename "grep") (f . arguments) where
+checkGrepRe = CommandCheck (Basename "grep") check where
+    check cmd = f cmd (arguments cmd)
     -- --regex=*(extglob) doesn't work. Fixme?
     skippable (Just s) = not ("--regex=" `isPrefixOf` s) && "-" `isPrefixOf` s
     skippable _ = False
-    f [] = return ()
-    f (x:r) | skippable (getLiteralStringExt (const $ return "_") x) = f r
-    f (re:_) = do
+    f _ [] = return ()
+    f cmd (x:r) | skippable (getLiteralStringExt (const $ return "_") x) = f cmd r
+    f cmd (re:_) = do
         when (isGlob re) $
             warn (getId re) 2062 "Quote the grep pattern so the shell won't interpret it."
-        let string = concat $ oversimplify re
-        if isConfusedGlobRegex string then
-            warn (getId re) 2063 "Grep uses regex, but this looks like a glob."
-          else potentially $ do
-            char <- getSuspiciousRegexWildcard string
-            return $ info (getId re) 2022 $
-                "Note that unlike globs, " ++ [char] ++ "* here matches '" ++ [char, char, char] ++ "' but not '" ++ wordStartingWith char ++ "'."
+
+        unless (cmd `hasFlag` "F") $ do
+            let string = concat $ oversimplify re
+            if isConfusedGlobRegex string then
+                warn (getId re) 2063 "Grep uses regex, but this looks like a glob."
+              else potentially $ do
+                char <- getSuspiciousRegexWildcard string
+                return $ info (getId re) 2022 $
+                    "Note that unlike globs, " ++ [char] ++ "* here matches '" ++ [char, char, char] ++ "' but not '" ++ wordStartingWith char ++ "'."
 
     wordStartingWith c =
         head . filter ([c] `isPrefixOf`) $ candidates
@@ -342,7 +343,7 @@ checkInjectableFindSh = CommandCheck (Basename "find") (check . arguments)
 
     pattern = [
         (`elem` ["-exec", "-execdir"]),
-        (`elem` ["sh", "bash", "ksh"]),
+        (`elem` ["sh", "bash", "dash", "ksh"]),
         (== "-c")
         ]
     action (id, arg) =
@@ -362,7 +363,7 @@ checkFindActionPrecedence = CommandCheck (Basename "find") (f . arguments)
         then warnFor (list !! (length pattern - 1))
         else f rest
     isMatch = isParam [ "-name", "-regex", "-iname", "-iregex", "-wholename", "-iwholename" ]
-    isAction = isParam [ "-exec", "-execdir", "-delete", "-print", "-print0" ]
+    isAction = isParam [ "-exec", "-execdir", "-delete", "-print", "-print0", "-fls", "-fprint", "-fprint0", "-fprintf", "-ls", "-ok", "-okdir", "-printf" ]
     isParam strs t = fromMaybe False $ do
         param <- getLiteralString t
         return $ param `elem` strs
@@ -470,16 +471,51 @@ prop_checkPrintfVar1 = verify checkPrintfVar "printf \"Lol: $s\""
 prop_checkPrintfVar2 = verifyNot checkPrintfVar "printf 'Lol: $s'"
 prop_checkPrintfVar3 = verify checkPrintfVar "printf -v cow $(cmd)"
 prop_checkPrintfVar4 = verifyNot checkPrintfVar "printf \"%${count}s\" var"
+prop_checkPrintfVar5 = verify checkPrintfVar "printf '%s %s %s' foo bar"
+prop_checkPrintfVar6 = verify checkPrintfVar "printf foo bar baz"
+prop_checkPrintfVar7 = verify checkPrintfVar "printf -- foo bar baz"
+prop_checkPrintfVar8 = verifyNot checkPrintfVar "printf '%s %s %s' \"${var[@]}\""
+prop_checkPrintfVar9 = verifyNot checkPrintfVar "printf '%s %s %s\\n' *.png"
+prop_checkPrintfVar10= verifyNot checkPrintfVar "printf '%s %s %s' foo bar baz"
+prop_checkPrintfVar11= verifyNot checkPrintfVar "printf '%(%s%s)T' -1"
 checkPrintfVar = CommandCheck (Exactly "printf") (f . arguments) where
+    f (doubledash:rest) | getLiteralString doubledash == Just "--" = f rest
     f (dashv:var:rest) | getLiteralString dashv == Just "-v" = f rest
-    f (format:params) = check format
+    f (format:params) = check format params
     f _ = return ()
-    check format =
+
+    countFormats string =
+        case string of
+            '%':'%':rest -> countFormats rest
+            '%':'(':rest -> 1 + countFormats (dropWhile (/= ')') rest)
+            '%':rest -> 1 + countFormats rest
+            _:rest -> countFormats rest
+            [] -> 0
+
+    check format more = do
+        fromMaybe (return ()) $ do
+            string <- getLiteralString format
+            let vars = countFormats string
+
+            return $ do
+                when (vars == 0 && more /= []) $
+                    err (getId format) 2182
+                        "This printf format string has no variables. Other arguments are ignored."
+
+                when (vars > 0
+                        && length more < vars
+                        && all (not . mayBecomeMultipleArgs) more) $
+                    warn (getId format) 2183 $
+                        "This format string has " ++ show vars ++ " variables, but is passed " ++ show (length more) ++ " arguments."
+
+
         unless ('%' `elem` concat (oversimplify format) || isLiteral format) $
-          warn (getId format) 2059
+          info (getId format) 2059
               "Don't use variables in the printf format string. Use printf \"..%s..\" \"$foo\"."
 
 
+
+
 prop_checkUuoeCmd1 = verify checkUuoeCmd "echo $(date)"
 prop_checkUuoeCmd2 = verify checkUuoeCmd "echo `date`"
 prop_checkUuoeCmd3 = verify checkUuoeCmd "echo \"$(date)\""
@@ -556,5 +592,94 @@ checkAliasesExpandEarly = CommandCheck (Exactly "alias") (f . arguments)
     checkArg _ = return ()
 
 
+prop_checkUnsetGlobs1 = verify checkUnsetGlobs "unset foo[1]"
+prop_checkUnsetGlobs2 = verifyNot checkUnsetGlobs "unset foo"
+checkUnsetGlobs = CommandCheck (Exactly "unset") (mapM_ check . arguments)
+  where
+    check arg =
+        when (isGlob arg) $
+            warn (getId arg) 2184 "Quote arguments to unset so they're not glob expanded."
+
+
+prop_checkFindWithoutPath1 = verify checkFindWithoutPath "find -type f"
+prop_checkFindWithoutPath2 = verify checkFindWithoutPath "find"
+prop_checkFindWithoutPath3 = verifyNot checkFindWithoutPath "find . -type f"
+prop_checkFindWithoutPath4 = verifyNot checkFindWithoutPath "find -H -L \"$path\" -print"
+checkFindWithoutPath = CommandCheck (Basename "find") f
+  where
+    f (T_SimpleCommand _ _ (cmd:args)) =
+        unless (hasPath args) $
+            info (getId cmd) 2185 "Some finds don't have a default path. Specify '.' explicitly."
+
+    -- This is a bit of a kludge. find supports flag arguments both before and after the path,
+    -- as well as multiple non-flag arguments that are not the path. We assume that all the
+    -- pre-path flags are single characters, which is generally the case.
+    hasPath (first:rest) =
+        let flag = fromJust $ getLiteralStringExt (const $ return "___") first in
+            not ("-" `isPrefixOf` flag) || length flag <= 2 && hasPath rest
+    hasPath [] = False
+
+
+prop_checkTimeParameters1 = verify checkTimeParameters "time -f lol sleep 10"
+prop_checkTimeParameters2 = verifyNot checkTimeParameters "time sleep 10"
+prop_checkTimeParameters3 = verifyNot checkTimeParameters "time -p foo"
+prop_checkTimeParameters4 = verifyNot checkTimeParameters "command time -f lol sleep 10"
+checkTimeParameters = CommandCheck (Exactly "time") f
+  where
+    f (T_SimpleCommand _ _ (cmd:args:_)) =
+        whenShell [Bash, Sh] $
+            let s = concat $ oversimplify args in
+                when ("-" `isPrefixOf` s && s /= "-p") $
+                    info (getId cmd) 2023 "The shell may override 'time' as seen in man time(1). Use 'command time ..' for that one."
+
+    f _ = return ()
+
+prop_checkTimedCommand1 = verify checkTimedCommand "#!/bin/sh\ntime -p foo | bar"
+prop_checkTimedCommand2 = verify checkTimedCommand "#!/bin/dash\ntime ( foo; bar; )"
+prop_checkTimedCommand3 = verifyNot checkTimedCommand "#!/bin/sh\ntime sleep 1"
+checkTimedCommand = CommandCheck (Exactly "time") f where
+    f (T_SimpleCommand _ _ (c:args@(_:_))) =
+        whenShell [Sh, Dash] $ do
+            let cmd = last args -- "time" is parsed with a command as argument
+            when (isPiped cmd) $
+                warn (getId c) 2176 "'time' is undefined for pipelines. time single stage or bash -c instead."
+            when (isSimple cmd == Just False) $
+                warn (getId cmd) 2177 "'time' is undefined for compound commands, time sh -c instead."
+    f _ = return ()
+    isPiped cmd =
+        case cmd of
+            T_Pipeline _ _ (_:_:_) -> True
+            _ -> False
+    getCommand cmd =
+        case cmd of
+            T_Pipeline _ _ (T_Redirecting _ _ a : _) -> return a
+            _ -> fail ""
+    isSimple cmd = do
+        innerCommand <- getCommand cmd
+        case innerCommand of
+            T_SimpleCommand {} -> return True
+            _ -> return False
+
+prop_checkLocalScope1 = verify checkLocalScope "local foo=3"
+prop_checkLocalScope2 = verifyNot checkLocalScope "f() { local foo=3; }"
+checkLocalScope = CommandCheck (Exactly "local") $ \t ->
+    whenShell [Bash, Dash] $ do -- Ksh allows it, Sh doesn't support local
+        path <- getPathM t
+        unless (any isFunction path) $
+            err (getId t) 2168 "'local' is only valid in functions."
+
+prop_checkDeprecatedTempfile1 = verify checkDeprecatedTempfile "var=$(tempfile)"
+prop_checkDeprecatedTempfile2 = verifyNot checkDeprecatedTempfile "tempfile=$(mktemp)"
+checkDeprecatedTempfile = CommandCheck (Basename "tempfile") $
+    \t -> warn (getId t) 2186 "tempfile is deprecated. Use mktemp instead."
+
+prop_checkDeprecatedEgrep = verify checkDeprecatedEgrep "egrep '.+'"
+checkDeprecatedEgrep = CommandCheck (Basename "egrep") $
+    \t -> info (getId t) 2196 "egrep is non-standard and deprecated. Use grep -E instead."
+
+prop_checkDeprecatedFgrep = verify checkDeprecatedFgrep "fgrep '*' files"
+checkDeprecatedFgrep = CommandCheck (Basename "fgrep") $
+    \t -> info (getId t) 2197 "fgrep is non-standard and deprecated. Use grep -F instead."
+
 return []
 runTests =  $( [| $(forAllProperties) (quickCheckWithResult (stdArgs { maxSuccess = 1 }) ) |])

ShellCheck/Checks/ShellSupport.hs

@@ -0,0 +1,388 @@
+{-
+    Copyright 2012-2016 Vidar Holen
+
+    This file is part of ShellCheck.
+    http://www.vidarholen.net/contents/shellcheck
+
+    ShellCheck is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    ShellCheck is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+-}
+{-# LANGUAGE TemplateHaskell #-}
+{-# LANGUAGE FlexibleContexts #-}
+module ShellCheck.Checks.ShellSupport (checker
+    , ShellCheck.Checks.ShellSupport.runTests
+) where
+
+import ShellCheck.AST
+import ShellCheck.ASTLib
+import ShellCheck.AnalyzerLib
+import ShellCheck.Interface
+import ShellCheck.Regex
+
+import Control.Monad
+import Control.Monad.RWS
+import Data.Char
+import Data.List
+import Data.Maybe
+import qualified Data.Map as Map
+import Test.QuickCheck.All (forAllProperties)
+import Test.QuickCheck.Test (quickCheckWithResult, stdArgs, maxSuccess)
+
+data ForShell = ForShell [Shell] (Token -> Analysis)
+
+getChecker params list = Checker {
+        perScript = nullCheck,
+        perToken = foldl composeAnalyzers nullCheck $ mapMaybe include list
+    }
+  where
+    shell = shellType params
+    include (ForShell list a) = do
+        guard $ shell `elem` list
+        return a
+
+checker params = getChecker params checks
+
+checks = [
+    checkForDecimals
+    ,checkBashisms
+    ,checkEchoSed
+    ,checkBraceExpansionVars
+    ,checkMultiDimensionalArrays
+    ]
+
+testChecker (ForShell _ t) =
+    Checker {
+        perScript = nullCheck,
+        perToken = t
+    }
+verify c s = producesComments (testChecker c) s == Just True
+verifyNot c s = producesComments (testChecker c) s == Just False
+
+prop_checkForDecimals1 = verify checkForDecimals "((3.14*c))"
+prop_checkForDecimals2 = verify checkForDecimals "foo[1.2]=bar"
+prop_checkForDecimals3 = verifyNot checkForDecimals "declare -A foo; foo[1.2]=bar"
+checkForDecimals = ForShell [Sh, Dash, Bash] f
+  where
+    f t@(TA_Expansion id _) = potentially $ do
+        str <- getLiteralString t
+        first <- str !!! 0
+        guard $ isDigit first && '.' `elem` str
+        return $ err id 2079 "(( )) doesn't support decimals. Use bc or awk."
+    f _ = return ()
+
+
+prop_checkBashisms = verify checkBashisms "while read a; do :; done < <(a)"
+prop_checkBashisms2 = verify checkBashisms "[ foo -nt bar ]"
+prop_checkBashisms3 = verify checkBashisms "echo $((i++))"
+prop_checkBashisms4 = verify checkBashisms "rm !(*.hs)"
+prop_checkBashisms5 = verify checkBashisms "source file"
+prop_checkBashisms6 = verify checkBashisms "[ \"$a\" == 42 ]"
+prop_checkBashisms7 = verify checkBashisms "echo ${var[1]}"
+prop_checkBashisms8 = verify checkBashisms "echo ${!var[@]}"
+prop_checkBashisms9 = verify checkBashisms "echo ${!var*}"
+prop_checkBashisms10= verify checkBashisms "echo ${var:4:12}"
+prop_checkBashisms11= verifyNot checkBashisms "echo ${var:-4}"
+prop_checkBashisms12= verify checkBashisms "echo ${var//foo/bar}"
+prop_checkBashisms13= verify checkBashisms "exec -c env"
+prop_checkBashisms14= verify checkBashisms "echo -n \"Foo: \""
+prop_checkBashisms15= verify checkBashisms "let n++"
+prop_checkBashisms16= verify checkBashisms "echo $RANDOM"
+prop_checkBashisms17= verify checkBashisms "echo $((RANDOM%6+1))"
+prop_checkBashisms18= verify checkBashisms "foo &> /dev/null"
+prop_checkBashisms19= verify checkBashisms "foo > file*.txt"
+prop_checkBashisms20= verify checkBashisms "read -ra foo"
+prop_checkBashisms21= verify checkBashisms "[ -a foo ]"
+prop_checkBashisms22= verifyNot checkBashisms "[ foo -a bar ]"
+prop_checkBashisms23= verify checkBashisms "trap mything ERR INT"
+prop_checkBashisms24= verifyNot checkBashisms "trap mything INT TERM"
+prop_checkBashisms25= verify checkBashisms "cat < /dev/tcp/host/123"
+prop_checkBashisms26= verify checkBashisms "trap mything ERR SIGTERM"
+prop_checkBashisms27= verify checkBashisms "echo *[^0-9]*"
+prop_checkBashisms28= verify checkBashisms "exec {n}>&2"
+prop_checkBashisms29= verify checkBashisms "echo ${!var}"
+prop_checkBashisms30= verify checkBashisms "printf -v '%s' \"$1\""
+prop_checkBashisms31= verify checkBashisms "printf '%q' \"$1\""
+prop_checkBashisms32= verifyNot checkBashisms "#!/bin/dash\n[ foo -nt bar ]"
+prop_checkBashisms33= verify checkBashisms "#!/bin/sh\necho -n foo"
+prop_checkBashisms34= verifyNot checkBashisms "#!/bin/dash\necho -n foo"
+prop_checkBashisms35= verifyNot checkBashisms "#!/bin/dash\nlocal foo"
+prop_checkBashisms36= verifyNot checkBashisms "#!/bin/dash\nread -p foo -r bar"
+prop_checkBashisms37= verifyNot checkBashisms "HOSTNAME=foo; echo $HOSTNAME"
+prop_checkBashisms38= verify checkBashisms "RANDOM=9; echo $RANDOM"
+prop_checkBashisms39= verify checkBashisms "foo-bar() { true; }"
+prop_checkBashisms40= verify checkBashisms "echo $(<file)"
+prop_checkBashisms41= verify checkBashisms "echo `<file`"
+prop_checkBashisms42= verify checkBashisms "trap foo int"
+prop_checkBashisms43= verify checkBashisms "trap foo sigint"
+prop_checkBashisms44= verifyNot checkBashisms "#!/bin/dash\ntrap foo int"
+prop_checkBashisms45= verifyNot checkBashisms "#!/bin/dash\ntrap foo INT"
+prop_checkBashisms46= verify checkBashisms "#!/bin/dash\ntrap foo SIGINT"
+prop_checkBashisms47= verify checkBashisms "#!/bin/dash\necho foo 42>/dev/null"
+prop_checkBashisms48= verifyNot checkBashisms "#!/bin/dash\necho $LINENO"
+prop_checkBashisms49= verify checkBashisms "#!/bin/dash\necho $MACHTYPE"
+prop_checkBashisms50= verify checkBashisms "#!/bin/sh\ncmd >& file"
+prop_checkBashisms51= verifyNot checkBashisms "#!/bin/sh\ncmd 2>&1"
+prop_checkBashisms52= verifyNot checkBashisms "#!/bin/sh\ncmd >&2"
+prop_checkBashisms53= verifyNot checkBashisms "#!/bin/sh\nprintf -- -f\n"
+prop_checkBashisms54= verify checkBashisms "#!/bin/sh\nfoo+=bar"
+checkBashisms = ForShell [Sh, Dash] $ \t -> do
+    params <- ask
+    kludge params t
+ where
+  -- This code was copy-pasted from Analytics where params was a variable
+  kludge params = bashism
+   where
+    isDash = shellType params == Dash
+    warnMsg id s =
+        if isDash
+        then warn id 2169 $ "In dash, " ++ s ++ " not supported."
+        else warn id 2039 $ "In POSIX sh, " ++ s ++ " undefined."
+
+    bashism (T_ProcSub id _ _) = warnMsg id "process substitution is"
+    bashism (T_Extglob id _ _) = warnMsg id "extglob is"
+    bashism (T_DollarSingleQuoted id _) = warnMsg id "$'..' is"
+    bashism (T_DollarDoubleQuoted id _) = warnMsg id "$\"..\" is"
+    bashism (T_ForArithmetic id _ _ _ _) = warnMsg id "arithmetic for loops are"
+    bashism (T_Arithmetic id _) = warnMsg id "standalone ((..)) is"
+    bashism (T_DollarBracket id _) = warnMsg id "$[..] in place of $((..)) is"
+    bashism (T_SelectIn id _ _ _) = warnMsg id "select loops are"
+    bashism (T_BraceExpansion id _) = warnMsg id "brace expansion is"
+    bashism (T_Condition id DoubleBracket _) = warnMsg id "[[ ]] is"
+    bashism (T_HereString id _) = warnMsg id "here-strings are"
+    bashism (TC_Binary id SingleBracket op _ _)
+        | op `elem` [ "-nt", "-ef", "\\<", "\\>"] =
+            unless isDash $ warnMsg id $ op ++ " is"
+    bashism (TC_Binary id SingleBracket "==" _ _) =
+            warnMsg id "== in place of = is"
+    bashism (TC_Unary id _ "-a" _) =
+            warnMsg id "unary -a in place of -e is"
+    bashism (TA_Unary id op _)
+        | op `elem` [ "|++", "|--", "++|", "--|"] =
+            warnMsg id $ filter (/= '|') op ++ " is"
+    bashism (TA_Binary id "**" _ _) = warnMsg id "exponentials are"
+    bashism (T_FdRedirect id "&" (T_IoFile _ (T_Greater _) _)) = warnMsg id "&> is"
+    bashism (T_FdRedirect id "" (T_IoFile _ (T_GREATAND _) _)) = warnMsg id ">& is"
+    bashism (T_FdRedirect id ('{':_) _) = warnMsg id "named file descriptors are"
+    bashism (T_FdRedirect id num _)
+        | all isDigit num && length num > 1 = warnMsg id "FDs outside 0-9 are"
+    bashism (T_Assignment id Append _ _ _) =
+        warnMsg id "+= is"
+    bashism (T_IoFile id _ word) | isNetworked =
+            warnMsg id "/dev/{tcp,udp} is"
+        where
+            file = onlyLiteralString word
+            isNetworked = any (`isPrefixOf` file) ["/dev/tcp", "/dev/udp"]
+    bashism (T_Glob id str) | "[^" `isInfixOf` str =
+            warnMsg id "^ in place of ! in glob bracket expressions is"
+
+    bashism t@(TA_Expansion id _) | isBashism =
+        warnMsg id $ fromJust str ++ " is"
+      where
+        str = getLiteralString t
+        isBashism = isJust str && isBashVariable (fromJust str)
+    bashism t@(T_DollarBraced id token) = do
+        mapM_ check expansion
+        when (isBashVariable var) $
+                    warnMsg id $ var ++ " is"
+      where
+        str = bracedString t
+        var = getBracedReference str
+        check (regex, feature) =
+            when (isJust $ matchRegex regex str) $ warnMsg id feature
+
+    bashism t@(T_Pipe id "|&") =
+        warnMsg id "|& in place of 2>&1 | is"
+    bashism (T_Array id _) =
+        warnMsg id "arrays are"
+    bashism (T_IoFile id _ t) | isGlob t =
+        warnMsg id "redirecting to/from globs is"
+    bashism (T_CoProc id _ _) =
+        warnMsg id "coproc is"
+
+    bashism (T_Function id _ _ str _) | not (isVariableName str) =
+        warnMsg id "naming functions outside [a-zA-Z_][a-zA-Z0-9_]* is"
+
+    bashism (T_DollarExpansion id [x]) | isOnlyRedirection x =
+        warnMsg id "$(<file) to read files is"
+    bashism (T_Backticked id [x]) | isOnlyRedirection x =
+        warnMsg id "`<file` to read files is"
+
+    bashism t@(T_SimpleCommand _ _ (cmd:arg:_))
+        | t `isCommand` "echo" && "-" `isPrefixOf` argString =
+            unless ("--" `isPrefixOf` argString) $ -- echo "-----"
+                if isDash
+                then
+                    when (argString /= "-n") $
+                        warnMsg (getId arg) "echo flags besides -n"
+                else
+                    warnMsg (getId arg) "echo flags are"
+      where argString = concat $ oversimplify arg
+    bashism t@(T_SimpleCommand _ _ (cmd:arg:_))
+        | t `isCommand` "exec" && "-" `isPrefixOf` concat (oversimplify arg) =
+            warnMsg (getId arg) "exec flags are"
+    bashism t@(T_SimpleCommand id _ _)
+        | t `isCommand` "let" = warnMsg id "'let' is"
+
+    bashism t@(T_SimpleCommand id _ (cmd:rest)) =
+        let name = fromMaybe "" $ getCommandName t
+            flags = getLeadingFlags t
+        in do
+            when (name `elem` unsupportedCommands) $
+                warnMsg id $ "'" ++ name ++ "' is"
+            potentially $ do
+                allowed <- Map.lookup name allowedFlags
+                (word, flag) <- listToMaybe $
+                    filter (\x -> (not . null . snd $ x) && snd x `notElem` allowed) flags
+                return . warnMsg (getId word) $ name ++ " -" ++ flag ++ " is"
+
+            when (name == "source") $ warnMsg id "'source' in place of '.' is"
+            when (name == "trap") $
+                let
+                    check token = potentially $ do
+                        str <- getLiteralString token
+                        let upper = map toUpper str
+                        return $ do
+                            when (upper `elem` ["ERR", "DEBUG", "RETURN"]) $
+                                warnMsg (getId token) $ "trapping " ++ str ++ " is"
+                            when ("SIG" `isPrefixOf` upper) $
+                                warnMsg (getId token)
+                                    "prefixing signal names with 'SIG' is"
+                            when (not isDash && upper /= str) $
+                                warnMsg (getId token)
+                                    "using lower/mixed case for signal names is"
+                in
+                    mapM_ check (drop 1 rest)
+
+            when (name == "printf") $ potentially $ do
+                format <- rest !!! 0  -- flags are covered by allowedFlags
+                let literal = onlyLiteralString format
+                guard $ "%q" `isInfixOf` literal
+                return $ warnMsg (getId format) "printf %q is"
+      where
+        unsupportedCommands = [
+            "let", "caller", "builtin", "complete", "compgen", "declare", "dirs", "disown",
+            "enable", "mapfile", "readarray", "pushd", "popd", "shopt", "suspend",
+            "typeset"
+            ] ++ if not isDash then ["local", "type"] else []
+        allowedFlags = Map.fromList [
+            ("read", if isDash then ["r", "p"] else ["r"]),
+            ("ulimit", ["f"]),
+            ("printf", []),
+            ("exec", [])
+            ]
+
+    bashism _ = return ()
+
+    varChars="_0-9a-zA-Z"
+    expansion = let re = mkRegex in [
+        (re $ "^![" ++ varChars ++ "]", "indirect expansion is"),
+        (re $ "^[" ++ varChars ++ "]+\\[.*\\]$", "array references are"),
+        (re $ "^![" ++ varChars ++ "]+\\[[*@]]$", "array key expansion is"),
+        (re $ "^![" ++ varChars ++ "]+[*@]$", "name matching prefixes are"),
+        (re $ "^[" ++ varChars ++ "]+:[^-=?+]", "string indexing is"),
+        (re $ "^[" ++ varChars ++ "]+(\\[.*\\])?/", "string replacement is")
+        ]
+    bashVars = [
+        "LINENO", "OSTYPE", "MACHTYPE", "HOSTTYPE", "HOSTNAME",
+        "DIRSTACK", "EUID", "UID", "SHLVL", "PIPESTATUS", "SHELLOPTS"
+        ]
+    bashDynamicVars = [ "RANDOM", "SECONDS" ]
+    dashVars = [ "LINENO" ]
+    isBashVariable var =
+        (var `elem` bashDynamicVars
+            || var `elem` bashVars && not (isAssigned var))
+        && not (isDash && var `elem` dashVars)
+    isAssigned var = any f (variableFlow params)
+      where
+        f x = case x of
+                Assignment (_, _, name, _) -> name == var
+                _ -> False
+
+prop_checkEchoSed1 = verify checkEchoSed "FOO=$(echo \"$cow\" | sed 's/foo/bar/g')"
+prop_checkEchoSed2 = verify checkEchoSed "rm $(echo $cow | sed -e 's,foo,bar,')"
+checkEchoSed = ForShell [Bash, Ksh] f
+  where
+    f (T_Pipeline id _ [a, b]) =
+        when (acmd == ["echo", "${VAR}"]) $
+            case bcmd of
+                ["sed", v] -> checkIn v
+                ["sed", "-e", v] -> checkIn v
+                _ -> return ()
+      where
+        -- This should have used backreferences, but TDFA doesn't support them
+        sedRe = mkRegex "^s(.)([^\n]*)g?$"
+        isSimpleSed s = fromMaybe False $ do
+            [first,rest] <- matchRegex sedRe s
+            let delimiters = filter (== head first) rest
+            guard $ length delimiters == 2
+            return True
+
+        acmd = oversimplify a
+        bcmd = oversimplify b
+        checkIn s =
+            when (isSimpleSed s) $
+                style id 2001 "See if you can use ${variable//search/replace} instead."
+    f _ = return ()
+
+
+prop_checkBraceExpansionVars1 = verify checkBraceExpansionVars "echo {1..$n}"
+prop_checkBraceExpansionVars2 = verifyNot checkBraceExpansionVars "echo {1,3,$n}"
+prop_checkBraceExpansionVars3 = verify checkBraceExpansionVars "eval echo DSC{0001..$n}.jpg"
+prop_checkBraceExpansionVars4 = verify checkBraceExpansionVars "echo {$i..100}"
+checkBraceExpansionVars = ForShell [Bash] f
+  where
+    f t@(T_BraceExpansion id list) = mapM_ check list
+      where
+        check element =
+            when (any (`isInfixOf` toString element) ["$..", "..$"]) $ do
+                c <- isEvaled element
+                if c
+                    then style id 2175 "Quote this invalid brace expansion since it should be passed literally to eval."
+                    else warn id 2051 "Bash doesn't support variables in brace range expansions."
+    f _ = return ()
+
+    literalExt t =
+        case t of
+            T_DollarBraced {} -> return "$"
+            T_DollarExpansion {} -> return "$"
+            T_DollarArithmetic {} -> return "$"
+            otherwise -> return "-"
+    toString t = fromJust $ getLiteralStringExt literalExt t
+    isEvaled t = do
+        cmd <- getClosestCommandM t
+        return $ isJust cmd && fromJust cmd `isUnqualifiedCommand` "eval"
+
+
+prop_checkMultiDimensionalArrays1 = verify checkMultiDimensionalArrays "foo[a][b]=3"
+prop_checkMultiDimensionalArrays2 = verifyNot checkMultiDimensionalArrays "foo[a]=3"
+prop_checkMultiDimensionalArrays3 = verify checkMultiDimensionalArrays "foo=( [a][b]=c )"
+prop_checkMultiDimensionalArrays4 = verifyNot checkMultiDimensionalArrays "foo=( [a]=c )"
+prop_checkMultiDimensionalArrays5 = verify checkMultiDimensionalArrays "echo ${foo[bar][baz]}"
+prop_checkMultiDimensionalArrays6 = verifyNot checkMultiDimensionalArrays "echo ${foo[bar]}"
+checkMultiDimensionalArrays = ForShell [Bash] f
+  where
+    f token =
+        case token of
+            T_Assignment _ _ name (first:second:_) _ -> about second
+            T_IndexedElement _ (first:second:_) _ -> about second
+            T_DollarBraced {} ->
+                when (isMultiDim token) $ about token
+            _ -> return ()
+    about t = warn (getId t) 2180 "Bash does not support multidimensional arrays. Use 1D or associative arrays."
+
+    re = mkRegex "^\\[.*\\]\\[.*\\]"  -- Fixme, this matches ${foo:- [][]} and such as well
+    isMultiDim t = getBracedModifier (bracedString t) `matches` re
+
+
+return []
+runTests =  $( [| $(forAllProperties) (quickCheckWithResult (stdArgs { maxSuccess = 1 }) ) |])

ShellCheck/Data.hs

@@ -33,6 +33,9 @@ internalVariables = [
     -- Other
     "USER", "TZ", "TERM", "LOGNAME", "LD_LIBRARY_PATH", "LANGUAGE", "DISPLAY",
     "HOSTNAME", "KRB5CCNAME", "XAUTHORITY"
+
+    -- Ksh
+    , ".sh.version"
   ]
 
 variablesWithoutSpaces = [
@@ -82,12 +85,24 @@ sampleWords = [
     "zulu"
   ]
 
+binaryTestOps = [
+    "-nt", "-ot", "-ef", "==", "!=", "<=", ">=", "-eq", "-ne", "-lt", "-le",
+    "-gt", "-ge", "=~", ">", "<", "=", "\\<", "\\>", "\\<=", "\\>="
+  ]
+
+unaryTestOps = [
+    "!", "-a", "-b", "-c", "-d", "-e", "-f", "-g", "-h", "-L", "-k", "-p",
+    "-r", "-s", "-S", "-t", "-u", "-w", "-x", "-O", "-G", "-N", "-z", "-n",
+    "-o", "-v", "-R"
+  ]
+
 shellForExecutable :: String -> Maybe Shell
 shellForExecutable name =
     case name of
         "sh"    -> return Sh
         "bash"  -> return Bash
         "dash"  -> return Dash
+        "ash"   -> return Dash -- There's also a warning for this.
         "ksh"   -> return Ksh
         "ksh88" -> return Ksh
         "ksh93" -> return Ksh

ShellCheck/Formatter/Format.hs

@@ -30,13 +30,15 @@ data Formatter = Formatter {
     footer :: IO ()
 }
 
-lineNo (PositionedComment pos _) = posLine pos
-colNo  (PositionedComment pos _) = posColumn pos
-codeNo (PositionedComment _ (Comment _ code _)) = code
-messageText (PositionedComment _ (Comment _ _ t)) = t
+lineNo (PositionedComment pos _ _) = posLine pos
+endLineNo (PositionedComment _ end _) = posLine end
+colNo  (PositionedComment pos _ _) = posColumn pos
+endColNo  (PositionedComment _ end _) = posColumn end
+codeNo (PositionedComment _ _ (Comment _ code _)) = code
+messageText (PositionedComment _ _ (Comment _ _ t)) = t
 
 severityText :: PositionedComment -> String
-severityText (PositionedComment _ (Comment c _ _)) =
+severityText (PositionedComment _ _ (Comment c _ _)) =
     case c of
         ErrorC   -> "error"
         WarningC -> "warning"
@@ -48,12 +50,15 @@ makeNonVirtual comments contents =
     map fix comments
   where
     ls = lines contents
-    fix c@(PositionedComment pos comment) = PositionedComment pos {
-        posColumn =
-            if lineNo c > 0 && lineNo c <= fromIntegral (length ls)
-            then real (ls !! fromIntegral (lineNo c - 1)) 0 0 (colNo c)
-            else colNo c
+    fix c@(PositionedComment start end comment) = PositionedComment start {
+        posColumn = realignColumn lineNo colNo c
+    } end {
+        posColumn = realignColumn endLineNo endColNo c
     } comment
+    realignColumn lineNo colNo c =
+      if lineNo c > 0 && lineNo c <= fromIntegral (length ls)
+      then real (ls !! fromIntegral (lineNo c - 1)) 0 0 (colNo c)
+      else colNo c
     real _ r v target | target <= v = r
     real [] r v _ = r -- should never happen
     real ('\t':rest) r v target =

ShellCheck/Formatter/JSON.hs

@@ -37,10 +37,12 @@ format = do
     }
 
 instance JSON (PositionedComment) where
-  showJSON comment@(PositionedComment pos (Comment level code string)) = makeObj [
-      ("file", showJSON $ posFile pos),
-      ("line", showJSON $ posLine pos),
-      ("column", showJSON $ posColumn pos),
+  showJSON comment@(PositionedComment start end (Comment level code string)) = makeObj [
+      ("file", showJSON $ posFile start),
+      ("line", showJSON $ posLine start),
+      ("endLine", showJSON $ posLine end),
+      ("column", showJSON $ posColumn start),
+      ("endColumn", showJSON $ posColumn end),
       ("level", showJSON $ severityText comment),
       ("code", showJSON code),
       ("message", showJSON string)

ShellCheck/Interface.hs

@@ -94,7 +94,7 @@ data Position = Position {
 } deriving (Show, Eq)
 
 data Comment = Comment Severity Code String deriving (Show, Eq)
-data PositionedComment = PositionedComment Position Comment deriving (Show, Eq)
+data PositionedComment = PositionedComment Position Position Comment deriving (Show, Eq)
 data TokenComment = TokenComment Id Comment deriving (Show, Eq)
 
 data ColorOption =

ShellCheck/Parser.hs

@@ -58,17 +58,18 @@ linefeed = do
     c <- char '\n'
     readPendingHereDocs
     return c
-singleQuote = char '\'' <|> unicodeSingleQuote
-doubleQuote = char '"' <|> unicodeDoubleQuote
+singleQuote = char '\''
+doubleQuote = char '"'
 variableStart = upper <|> lower <|> oneOf "_"
 variableChars = upper <|> lower <|> digit <|> oneOf "_"
 functionChars = variableChars <|> oneOf ":+-.?"
 specialVariable = oneOf "@*#?-$!"
+paramSubSpecialChars = oneOf "/:+-=%"
 quotableChars = "|&;<>()\\ '\t\n\r\xA0" ++ doubleQuotableChars
-quotable = almostSpace <|> unicodeDoubleQuote <|> oneOf quotableChars
+quotable = almostSpace <|> oneOf quotableChars
 bracedQuotable = oneOf "}\"$`'"
-doubleQuotableChars = "\"$`" ++ unicodeDoubleQuoteChars
-doubleQuotable = unicodeDoubleQuote <|> oneOf doubleQuotableChars
+doubleQuotableChars = "\"$`"
+doubleQuotable = oneOf doubleQuotableChars
 whitespace = oneOf " \t" <|> carriageReturn <|> almostSpace <|> linefeed
 linewhitespace = oneOf " \t" <|> almostSpace
 
@@ -77,7 +78,8 @@ suspectCharAfterQuotes = variableChars <|> char '%'
 extglobStartChars = "?*@!+"
 extglobStart = oneOf extglobStartChars
 
-unicodeDoubleQuoteChars = "\x201C\x201D\x2033\x2036"
+unicodeDoubleQuotes = "\x201C\x201D\x2033\x2036"
+unicodeSingleQuotes = "\x2018\x2019"
 
 prop_spacing = isOk spacing "  \\\n # Comment"
 spacing = do
@@ -106,17 +108,12 @@ allspacingOrFail = do
     s <- allspacing
     when (null s) $ fail "Expected whitespace"
 
-unicodeDoubleQuote = do
+readUnicodeQuote = do
     pos <- getPosition
-    oneOf unicodeDoubleQuoteChars
-    parseProblemAt pos WarningC 1015 "This is a unicode double quote. Delete and retype it."
-    return '"'
-
-unicodeSingleQuote = do
-    pos <- getPosition
-    char '\x2018' <|> char '\x2019'
-    parseProblemAt pos WarningC 1016 "This is a unicode single quote. Delete and retype it."
-    return '"'
+    c <- oneOf (unicodeSingleQuotes ++ unicodeDoubleQuotes)
+    parseProblemAt pos WarningC 1110 "This is a unicode quote. Delete and retype it (or quote to make literal)."
+    id <- getNextIdAt pos
+    return $ T_Literal id [c]
 
 carriageReturn = do
     parseNote ErrorC 1017 "Literal carriage return. Run script through tr -d '\\r' ."
@@ -135,7 +132,7 @@ almostSpace =
 
 --------- Message/position annotation on top of user state
 data Note = Note Id Severity Code String deriving (Show, Eq)
-data ParseNote = ParseNote SourcePos Severity Code String deriving (Show, Eq)
+data ParseNote = ParseNote SourcePos SourcePos Severity Code String deriving (Show, Eq)
 data Context =
         ContextName SourcePos String
         | ContextAnnotation [Annotation]
@@ -162,9 +159,9 @@ initialUserState = UserState {
     pendingHereDocs = []
 }
 
-codeForParseNote (ParseNote _ _ code _) = code
+codeForParseNote (ParseNote _ _ _ code _) = code
 noteToParseNote map (Note id severity code message) =
-        ParseNote pos severity code message
+        ParseNote pos pos severity code message
     where
         pos = fromJust $ Map.lookup id map
 
@@ -181,6 +178,7 @@ getNextIdAt sourcepos = do
     return newId
   where incId (Id n) = Id $ n+1
 
+getNextId :: Monad m => SCParser m Id
 getNextId = do
     pos <- getPosition
     getNextIdAt pos
@@ -320,14 +318,23 @@ pushContext c = do
     v <- getCurrentContexts
     setCurrentContexts (c:v)
 
-parseProblemAt pos level code msg = do
+parseProblemAtWithEnd start end level code msg = do
     irrelevant <- shouldIgnoreCode code
     unless irrelevant $
         Ms.modify (\state -> state {
             parseProblems = note:parseProblems state
         })
   where
-    note = ParseNote pos level code msg
+    note = ParseNote start end level code msg
+
+parseProblemAt pos = parseProblemAtWithEnd pos pos
+
+parseProblemAtId :: Monad m => Id -> Severity -> Integer -> String -> SCParser m ()
+parseProblemAtId id level code msg = do
+    map <- getMap
+    let pos = Map.findWithDefault
+                (error "Internal error (no position for id). Please report.") id map
+    parseProblemAt pos level code msg
 
 -- Store non-parse problems inside
 
@@ -335,7 +342,9 @@ parseNote c l a = do
     pos <- getPosition
     parseNoteAt pos c l a
 
-parseNoteAt pos c l a = addParseNote $ ParseNote pos c l a
+parseNoteAt pos c l a = addParseNote $ ParseNote pos pos c l a
+
+parseNoteAtWithEnd start end c l a = addParseNote $ ParseNote start end c l a
 
 --------- Convenient combinators
 thenSkip main follow = do
@@ -406,7 +415,7 @@ readConditionContents single =
                                 pos <- getPosition
                                 s <- many1 letter
                                 when (s `elem` commonCommands) $
-                                    parseProblemAt pos WarningC 1009 "Use 'if cmd; then ..' to check exit code, or 'if [[ $(cmd) == .. ]]' to check output.")
+                                    parseProblemAt pos WarningC 1014 "Use 'if cmd; then ..' to check exit code, or 'if [[ $(cmd) == .. ]]' to check output.")
 
   where
     spacingOrLf = condSpacing True
@@ -442,7 +451,7 @@ readConditionContents single =
             c <- oneOf "'\""
             s <- anyOp
             char c
-            return s
+            return $ escaped s
 
         anyOp = flagOp <|> flaglessOp <|> fail
                     "Expected comparison operator (don't wrap commands in []/[[]])"
@@ -522,7 +531,7 @@ readConditionContents single =
         condSpacing requiresSpacing
         return x
 
-    readCondNoaryOrBinary = do
+    readCondNullaryOrBinary = do
       id <- getNextId
       x <- readCondWord `attempting` (do
               pos <- getPosition
@@ -539,7 +548,16 @@ readConditionContents single =
                     then readRegex
                     else  readCondWord <|> (parseProblemAt pos ErrorC 1027 "Expected another argument for this operator." >> mzero)
             return (x `op` y)
-          ) <|> return (TC_Noary id typ x)
+          ) <|> ( do
+            checkTrailingOp x
+            return $ TC_Nullary id typ x
+          )
+
+    checkTrailingOp x = fromMaybe (return ()) $ do
+        (T_Literal id str) <- getTrailingUnquotedLiteral x
+        trailingOp <- listToMaybe (filter (`isSuffixOf` str) binaryTestOps)
+        return $ parseProblemAtId id ErrorC 1108 $
+            "You need a space before and after the " ++ trailingOp ++ " ."
 
     readCondGroup = do
           id <- getNextId
@@ -616,7 +634,7 @@ readConditionContents single =
         return $ TC_Unary id typ "!" expr
 
     readCondExpr =
-      readCondGroup <|> readCondUnaryExp <|> readCondNoaryOrBinary
+      readCondGroup <|> readCondUnaryExp <|> readCondNullaryOrBinary
 
     readCondOr = chainl1 readCondAnd readCondAndOp
     readCondAnd = chainl1 readCondTerm readCondOrOp
@@ -645,6 +663,7 @@ prop_a19= isOk readArithmeticContents "\\\n3 +\\\n  2"
 prop_a20= isOk readArithmeticContents "a ? b ? c : d : e"
 prop_a21= isOk readArithmeticContents "a ? b : c ? d : e"
 prop_a22= isOk readArithmeticContents "!!a"
+readArithmeticContents :: Monad m => SCParser m Token
 readArithmeticContents =
     readSequence
   where
@@ -658,12 +677,40 @@ readArithmeticContents =
         id <- getNextId
         op <- choice (map (\x -> try $ do
                                         s <- string x
-                                        notFollowedBy2 $ oneOf "&|<>="
+                                        failIfIncompleteOp
                                         return s
                             ) op)
         spacing
         return $ token id op
 
+    failIfIncompleteOp = notFollowedBy2 $ oneOf "&|<>="
+
+    -- Read binary minus, but also check for -lt, -gt and friends:
+    readMinusOp = do
+        id <- getNextId
+        pos <- getPosition
+        try $ do
+            char '-'
+            failIfIncompleteOp
+        optional $ do
+            (str, alt) <- lookAhead . choice $ map tryOp [
+                ("lt", "<"),
+                ("gt", ">"),
+                ("le", "<="),
+                ("ge", ">="),
+                ("eq", "=="),
+                ("ne", "!=")
+              ]
+            parseProblemAt pos ErrorC 1106 $ "In arithmetic contexts, use " ++ alt ++ " instead of -" ++ str
+        spacing
+        return $ TA_Binary id "-"
+      where
+        tryOp (str, alt) = try $ do
+            string str
+            spacing1
+            return (str, alt)
+
+
     readArrayIndex = do
         id <- getNextId
         char '['
@@ -733,7 +780,7 @@ readArithmeticContents =
     readEquated = readCompared `splitBy` ["==", "!="]
     readCompared = readShift `splitBy` ["<=", ">=", "<", ">"]
     readShift = readAddition `splitBy` ["<<", ">>"]
-    readAddition = readMultiplication `splitBy` ["+", "-"]
+    readAddition = chainl1 readMultiplication (readBinary ["+"] <|> readMinusOp)
     readMultiplication = readExponential `splitBy` ["*", "/", "%"]
     readExponential = readAnyNegated `splitBy` ["**"]
 
@@ -810,7 +857,7 @@ readCondition = called "test expression" $ do
     pos <- getPosition
     space <- allspacing
     when (null space) $
-        parseProblemAt pos ErrorC 1035 $ "You need a space after the " ++
+        parseProblemAtWithEnd opos pos ErrorC 1035 $ "You need a space after the " ++
             if single
                 then "[ and before the ]."
                 else "[[ and before the ]]."
@@ -835,10 +882,11 @@ readAnnotationPrefix = do
 prop_readAnnotation1 = isOk readAnnotation "# shellcheck disable=1234,5678\n"
 prop_readAnnotation2 = isOk readAnnotation "# shellcheck disable=SC1234 disable=SC5678\n"
 prop_readAnnotation3 = isOk readAnnotation "# shellcheck disable=SC1234 source=/dev/null disable=SC5678\n"
+prop_readAnnotation4 = isWarning readAnnotation "# shellcheck cats=dogs disable=SC1234\n"
 readAnnotation = called "shellcheck annotation" $ do
     try readAnnotationPrefix
     many1 linewhitespace
-    values <- many1 (readDisable <|> readSourceOverride <|> readShellOverride)
+    values <- many1 (readDisable <|> readSourceOverride <|> readShellOverride <|> anyKey)
     linefeed
     many linewhitespace
     return $ concat values
@@ -870,6 +918,13 @@ readAnnotation = called "shellcheck annotation" $ do
         many linewhitespace
         return value
 
+    anyKey = do
+        pos <- getPosition
+        anyChar `reluctantlyTill1` whitespace
+        many linewhitespace
+        parseNoteAt pos WarningC 1107 "This directive is unknown. It will be ignored."
+        return []
+
 readAnnotations = do
     annotations <- many (readAnnotation `thenSkip` allspacing)
     return $ concat annotations
@@ -888,6 +943,9 @@ prop_readNormalWord6 = isOk readNormalWord "foo/{}"
 prop_readNormalWord7 = isOk readNormalWord "foo\\\nbar"
 prop_readNormalWord8 = isWarning readSubshell "(foo\\ \nbar)"
 prop_readNormalWord9 = isOk readSubshell "(foo\\ ;\nbar)"
+prop_readNormalWord10 = isWarning readNormalWord "\x201Chello\x201D"
+prop_readNormalWord11 = isWarning readNormalWord "\x2018hello\x2019"
+prop_readNormalWord12 = isWarning readNormalWord "hello\x2018"
 readNormalWord = readNormalishWord ""
 
 readNormalishWord end = do
@@ -897,6 +955,20 @@ readNormalishWord end = do
     checkPossibleTermination pos x
     return $ T_NormalWord id x
 
+readIndexSpan = do
+    id <- getNextId
+    x <- many (readNormalWordPart "]" <|> someSpace <|> otherLiteral)
+    return $ T_NormalWord id x
+  where
+    someSpace = do
+        id <- getNextId
+        str <- spacing1
+        return $ T_Literal id str
+    otherLiteral = do
+        id <- getNextId
+        str <- many1 $ oneOf quotableChars
+        return $ T_Literal id str
+
 checkPossibleTermination pos [T_Literal _ x] =
     when (x `elem` ["do", "done", "then", "fi", "esac"]) $
         parseProblemAt pos WarningC 1010 $ "Use semicolon or linefeed before '" ++ x ++ "' (or quote to make it literal)."
@@ -913,6 +985,7 @@ readNormalWordPart end = do
         readBraced,
         readUnquotedBackTicked,
         readProcSub,
+        readUnicodeQuote,
         readNormalLiteral end,
         readLiteralCurlyBraces
       ]
@@ -947,13 +1020,19 @@ readDollarBracedWord = do
     list <- many readDollarBracedPart
     return $ T_NormalWord id list
 
-readDollarBracedPart = readSingleQuoted <|> readDoubleQuoted <|> readExtglob <|> readNormalDollar <|> readUnquotedBackTicked <|> readDollarBracedLiteral
+readDollarBracedPart = readSingleQuoted <|> readDoubleQuoted <|>
+                       readParamSubSpecialChar <|> readExtglob <|> readNormalDollar <|>
+                       readUnquotedBackTicked <|> readDollarBracedLiteral
 
 readDollarBracedLiteral = do
     id <- getNextId
     vars <- (readBraceEscaped <|> (anyChar >>= \x -> return [x])) `reluctantlyTill1` bracedQuotable
     return $ T_Literal id $ concat vars
 
+readParamSubSpecialChar = do
+    id <- getNextId
+    T_ParamSubSpecialChar id <$> many1 paramSubSpecialChars
+
 prop_readProcSub1 = isOk readProcSub "<(echo test | wc -l)"
 prop_readProcSub2 = isOk readProcSub "<(  if true; then true; fi )"
 prop_readProcSub3 = isOk readProcSub "<( # nothing here \n)"
@@ -970,15 +1049,16 @@ readProcSub = called "process substitution" $ do
 
 prop_readSingleQuoted = isOk readSingleQuoted "'foo bar'"
 prop_readSingleQuoted2 = isWarning readSingleQuoted "'foo bar\\'"
-prop_readsingleQuoted3 = isWarning readSingleQuoted "\x2018hello\x2019"
 prop_readSingleQuoted4 = isWarning readNormalWord "'it's"
 prop_readSingleQuoted5 = isWarning readSimpleCommand "foo='bar\ncow 'arg"
 prop_readSingleQuoted6 = isOk readSimpleCommand "foo='bar cow 'arg"
+prop_readSingleQuoted7 = isOk readSingleQuoted "'foo\x201C\&bar'"
+prop_readSingleQuoted8 = isWarning readSingleQuoted "'foo\x2018\&bar'"
 readSingleQuoted = called "single quoted string" $ do
     id <- getNextId
     startPos <- getPosition
     singleQuote
-    s <- readSingleQuotedPart `reluctantlyTill` singleQuote
+    s <- many readSingleQuotedPart
     let string = concat s
     endPos <- getPosition
     singleQuote <|> fail "Expected end of single quoted string"
@@ -1003,7 +1083,15 @@ readSingleQuotedLiteral = do
 
 readSingleQuotedPart =
     readSingleEscaped
-    <|> many1 (noneOf "'\\\x2018\x2019")
+    <|> many1 (noneOf $ "'\\" ++ unicodeSingleQuotes)
+    <|> readUnicodeQuote
+   where
+    readUnicodeQuote = do
+        pos <- getPosition
+        x <- oneOf unicodeSingleQuotes
+        parseProblemAt pos WarningC 1112
+            "This is a unicode quote. Delete and retype it (or ignore/doublequote for literal)."
+        return [x]
 
 
 prop_readBackTicked = isOk (readBackTicked False) "`ls *.mp3`"
@@ -1060,13 +1148,18 @@ subParse pos parser input = do
     setPosition lastPosition
     return result
 
-inSeparateContext parser = do
+-- Parse something, but forget all parseProblems
+inSeparateContext = parseForgettingContext True
+-- Parse something, but forget all parseProblems on failure
+forgetOnFailure = parseForgettingContext False
+
+parseForgettingContext alsoOnSuccess parser = do
     context <- Ms.get
     success context <|> failure context
   where
     success c = do
         res <- try parser
-        Ms.put c
+        when alsoOnSuccess $ Ms.put c
         return res
     failure c = do
         Ms.put c
@@ -1074,11 +1167,12 @@ inSeparateContext parser = do
 
 prop_readDoubleQuoted = isOk readDoubleQuoted "\"Hello $FOO\""
 prop_readDoubleQuoted2 = isOk readDoubleQuoted "\"$'\""
-prop_readDoubleQuoted3 = isWarning readDoubleQuoted "\x201Chello\x201D"
+prop_readDoubleQuoted3 = isOk readDoubleQuoted "\"\x2018hello\x2019\""
 prop_readDoubleQuoted4 = isWarning readSimpleCommand "\"foo\nbar\"foo"
 prop_readDoubleQuoted5 = isOk readSimpleCommand "lol \"foo\nbar\" etc"
 prop_readDoubleQuoted6 = isOk readSimpleCommand "echo \"${ ls; }\""
 prop_readDoubleQuoted7 = isOk readSimpleCommand "echo \"${ ls;}bar\""
+prop_readDoubleQuoted8 = isWarning readDoubleQuoted "\"\x201Chello\x201D\""
 readDoubleQuoted = called "double quoted string" $ do
     id <- getNextId
     startPos <- getPosition
@@ -1103,7 +1197,15 @@ suggestForgotClosingQuote startPos endPos name = do
     parseProblemAt endPos InfoC 1079
         "This is actually an end quote, but due to next char it looks suspect."
 
-doubleQuotedPart = readDoubleLiteral <|> readDoubleQuotedDollar <|> readQuotedBackTicked
+doubleQuotedPart = readDoubleLiteral <|> readDoubleQuotedDollar <|> readQuotedBackTicked <|> readUnicodeQuote
+  where
+    readUnicodeQuote = do
+        pos <- getPosition
+        id <- getNextId
+        c <- oneOf unicodeDoubleQuotes
+        parseProblemAt pos WarningC 1111
+            "This is a unicode quote. Delete and retype it (or ignore/singlequote for literal)."
+        return $ T_Literal id [c]
 
 readDoubleQuotedLiteral = do
     doubleQuote
@@ -1117,7 +1219,7 @@ readDoubleLiteral = do
     return $ T_Literal id (concat s)
 
 readDoubleLiteralPart = do
-    x <- many1 (readDoubleEscaped <|> many1 (noneOf ('\\':doubleQuotableChars)))
+    x <- many1 (readDoubleEscaped <|> many1 (noneOf ('\\':doubleQuotableChars ++ unicodeDoubleQuotes)))
     return $ concat x
 
 readNormalLiteral end = do
@@ -1159,8 +1261,15 @@ readGlob = readExtglob <|> readSimple <|> readClass <|> readGlobbyLiteral
             c <- extglobStart <|> char '['
             return $ T_Literal id [c]
 
-readNormalLiteralPart end =
-    readNormalEscaped <|> many1 (noneOf (end ++ quotableChars ++ extglobStartChars ++ "[{}"))
+readNormalLiteralPart customEnd =
+    readNormalEscaped <|>
+        many1 (noneOf (customEnd ++ standardEnd))
+  where
+    standardEnd = "[{}"
+        ++ quotableChars
+        ++ extglobStartChars
+        ++ unicodeDoubleQuotes
+        ++ unicodeSingleQuotes
 
 readNormalEscaped = called "escaped char" $ do
     pos <- getPosition
@@ -1225,19 +1334,16 @@ readExtglobPart = do
 
 
 readSingleEscaped = do
+    pos <- getPosition
     s <- backslash
-    let attempt level code p msg = do { try $ parseNote level code msg; x <- p; return [s,x]; }
+    x <- lookAhead anyChar
 
-    do {
-        x <- lookAhead singleQuote;
-        parseProblem InfoC 1003 "Are you trying to escape that single quote? echo 'You'\\''re doing it wrong'.";
-        return [s];
-    }
-        <|> attempt InfoC 1004 linefeed "You don't break lines with \\ in single quotes, it results in literal backslash-linefeed."
-        <|> do
-            x <- anyChar
-            return [s,x]
+    case x of
+        '\'' -> parseProblemAt pos InfoC 1003 "Want to escape a single quote? echo 'This is how it'\''s done'.";
+        '\n' -> parseProblemAt pos InfoC 1004 "This backslash+linefeed is literal. Break outside single quotes if you just want to break the line."
+        _ -> return ()
 
+    return [s]
 
 readDoubleEscaped = do
     bs <- backslash
@@ -1298,12 +1404,32 @@ readBraced = try braceExpansion
     braceLiteral =
         T_Literal `withParser` readGenericLiteral1 (oneOf "{}\"$'," <|> whitespace)
 
-readNormalDollar = readDollarExpression <|> readDollarDoubleQuote <|> readDollarSingleQuote <|> readDollarLonely
-readDoubleQuotedDollar = readDollarExpression <|> readDollarLonely
+ensureDollar =
+    -- The grammar should have been designed along the lines of readDollarExpr = char '$' >> stuff, but
+    -- instead, each subunit parses its own $. This results in ~7 1-3 char lookaheads instead of one 1-char.
+    -- Instead of optimizing the grammar, here's a green cut that decreases shellcheck runtime by 10%:
+    lookAhead $ char '$'
+
+readNormalDollar = do
+    ensureDollar
+    readDollarExp <|> readDollarDoubleQuote <|> readDollarSingleQuote <|> readDollarLonely
+readDoubleQuotedDollar = do
+    ensureDollar
+    readDollarExp <|> readDollarLonely
+
 
 prop_readDollarExpression1 = isOk readDollarExpression "$(((1) && 3))"
 prop_readDollarExpression2 = isWarning readDollarExpression "$(((1)) && 3)"
-readDollarExpression = readTripleParenthesis "$" readDollarArithmetic readDollarExpansion <|> readDollarArithmetic <|> readDollarBracket <|> readDollarBraceCommandExpansion <|> readDollarBraced <|> readDollarExpansion <|> readDollarVariable
+prop_readDollarExpression3 = isWarning readDollarExpression "$((\"$@\" &); foo;)"
+readDollarExpression :: Monad m => SCParser m Token
+readDollarExpression = do
+    ensureDollar
+    readDollarExp
+
+readDollarExp = arithmetic <|> readDollarExpansion <|> readDollarBracket <|> readDollarBraceCommandExpansion <|> readDollarBraced <|> readDollarVariable
+  where
+    arithmetic = readAmbiguous "$((" readDollarArithmetic readDollarExpansion (\pos ->
+        parseNoteAt pos WarningC 1102 "Shells disambiguate $(( differently or not at all. For $(command substition), add space after $( . For $((arithmetics)), fix parsing errors.")
 
 prop_readDollarSingleQuote = isOk readDollarSingleQuote "$'foo\\\'lol'"
 readDollarSingleQuote = called "$'..' expression" $ do
@@ -1349,25 +1475,20 @@ readArithmeticExpression = called "((..)) command" $ do
     string "))"
     return (T_Arithmetic id c)
 
--- Check if maybe ((( was intended as ( (( rather than (( (
-readTripleParenthesis prefix expected alternative = do
-    pos <- try . lookAhead $ do
-        string prefix
-        p <- getPosition
-        string "(((" -- should optimally be "((" but it's noisy and rarely helpful
-        return p
-
+-- If the next characters match prefix, try two different parsers and warn if the alternate parser had to be used
+readAmbiguous :: Monad m => String -> SCParser m p -> SCParser m p -> (SourcePos -> SCParser m ()) -> SCParser m p
+readAmbiguous prefix expected alternative warner = do
+    pos <- getPosition
+    try . lookAhead $ string prefix
     -- If the expected parser fails, try the alt.
     -- If the alt fails, run the expected one again for the errors.
-    try expected <|> tryAlt pos <|> expected
+    try expected <|> try (withAlt pos) <|> expected
   where
-    tryAlt pos = do
-        t <- try alternative
-        parseNoteAt pos WarningC 1102 $
-            "Shells differ in parsing ambiguous " ++ prefix ++ "(((. Use spaces: " ++ prefix ++ "( (( ."
+    withAlt pos = do
+        t <- forgetOnFailure alternative
+        warner pos
         return t
 
-
 prop_readDollarBraceCommandExpansion1 = isOk readDollarBraceCommandExpansion "${ ls; }"
 prop_readDollarBraceCommandExpansion2 = isOk readDollarBraceCommandExpansion "${\nls\n}"
 readDollarBraceCommandExpansion = called "ksh ${ ..; } command expansion" $ do
@@ -1461,40 +1582,32 @@ prop_readHereDoc8 = isOk readScript "cat <<foo>>bar\netc\nfoo"
 prop_readHereDoc9 = isOk readScript "if true; then cat << foo; fi\nbar\nfoo\n"
 prop_readHereDoc10= isOk readScript "if true; then cat << foo << bar; fi\nfoo\nbar\n"
 prop_readHereDoc11= isOk readScript "cat << foo $(\nfoo\n)lol\nfoo\n"
+prop_readHereDoc12= isOk readScript "cat << foo|cat\nbar\nfoo"
 readHereDoc = called "here document" $ do
     fid <- getNextId
     pos <- getPosition
     try $ string "<<"
     dashed <- (char '-' >> return Dashed) <|> return Undashed
-    tokenPosition <- getPosition
     sp <- spacing
     optional $ do
         try . lookAhead $ char '('
         let message = "Shells are space sensitive. Use '< <(cmd)', not '<<" ++ sp ++ "(cmd)'."
         parseProblemAt pos ErrorC 1038 message
     hid <- getNextId
-    (quoted, endToken) <-
-            liftM (\ x -> (Quoted, stripLiteral x)) readDoubleQuotedLiteral
-            <|> liftM (\ x -> (Quoted, x)) readSingleQuotedLiteral
-            <|> (readToken >>= (\x -> return (Unquoted, x)))
+    (quoted, endToken) <- readToken
 
     -- add empty tokens for now, read the rest in readPendingHereDocs
     let doc = T_HereDoc hid dashed quoted endToken []
     addPendingHereDoc doc
-    return $ T_FdRedirect fid "" doc
+    return doc
   where
-    stripLiteral (T_Literal _ x) = x
-    stripLiteral (T_SingleQuoted _ x) = x
-
-    readToken =
-        liftM concat $ many1 (escaped <|> quoted <|> normal)
-      where
-        quoted = liftM stripLiteral readDoubleQuotedLiteral <|> readSingleQuotedLiteral
-        normal = anyChar `reluctantlyTill1` (whitespace <|> oneOf "<>;&)'\"\\")
-        escaped = do -- surely the user must be doing something wrong at this point
-            char '\\'
-            c <- anyChar
-            return [c]
+    quotes = "\"'\\"
+    -- Fun fact: bash considers << foo"" quoted, but not << <("foo").
+    -- Instead of replicating this, just read a token and strip quotes.
+    readToken = do
+        str <- readStringForParser readNormalWord
+        return (if any (`elem` quotes) str then Quoted else Unquoted,
+                filter (not . (`elem` quotes)) str)
 
 
 readPendingHereDocs = do
@@ -1552,7 +1665,13 @@ readPendingHereDocs = do
 
 
 readFilename = readNormalWord
-readIoFileOp = choice [g_LESSAND, g_GREATAND, g_DGREAT, g_LESSGREAT, g_CLOBBER, redirToken '<' T_Less, redirToken '>' T_Greater ]
+readIoFileOp = choice [g_DGREAT, g_LESSGREAT, g_GREATAND, g_LESSAND, g_CLOBBER, redirToken '<' T_Less, redirToken '>' T_Greater ]
+
+readIoDuplicate = try $ do
+    id <- getNextId
+    op <- g_GREATAND <|> g_LESSAND
+    target <- readIoVariable <|> many1 digit <|> string "-"
+    return $ T_IoDuplicate id op target
 
 prop_readIoFile = isOk readIoFile ">> \"$(date +%YYmmDD)\""
 readIoFile = called "redirection" $ do
@@ -1560,35 +1679,31 @@ readIoFile = called "redirection" $ do
     op <- readIoFileOp
     spacing
     file <- readFilename
-    return $ T_FdRedirect id "" $ T_IoFile id op file
+    return $ T_IoFile id op file
 
 readIoVariable = try $ do
     char '{'
     x <- readVariableName
     char '}'
-    lookAhead readIoFileOp
     return $ "{" ++ x ++ "}"
 
-readIoNumber = try $ do
-    x <- many1 digit <|> string "&"
-    lookAhead readIoFileOp
+readIoSource = try $ do
+    x <- string "&" <|> readIoVariable <|> many digit
+    lookAhead $ void readIoFileOp <|> void (string "<<")
     return x
 
-prop_readIoNumberRedirect = isOk readIoNumberRedirect "3>&2"
-prop_readIoNumberRedirect2 = isOk readIoNumberRedirect "2> lol"
-prop_readIoNumberRedirect3 = isOk readIoNumberRedirect "4>&-"
-prop_readIoNumberRedirect4 = isOk readIoNumberRedirect "&> lol"
-prop_readIoNumberRedirect5 = isOk readIoNumberRedirect "{foo}>&2"
-prop_readIoNumberRedirect6 = isOk readIoNumberRedirect "{foo}<&-"
-readIoNumberRedirect = do
+prop_readIoRedirect = isOk readIoRedirect "3>&2"
+prop_readIoRedirect2 = isOk readIoRedirect "2> lol"
+prop_readIoRedirect3 = isOk readIoRedirect "4>&-"
+prop_readIoRedirect4 = isOk readIoRedirect "&> lol"
+prop_readIoRedirect5 = isOk readIoRedirect "{foo}>&2"
+prop_readIoRedirect6 = isOk readIoRedirect "{foo}<&-"
+readIoRedirect = do
     id <- getNextId
-    n <- readIoVariable <|> readIoNumber
-    op <- readHereString <|> readHereDoc <|> readIoFile
-    let actualOp = case op of T_FdRedirect _ "" x -> x
+    n <- readIoSource
+    redir <- readHereString <|> readHereDoc <|> readIoDuplicate <|> readIoFile
     spacing
-    return $ T_FdRedirect id n actualOp
-
-readIoRedirect = choice [ readIoNumberRedirect, readHereString, readHereDoc, readIoFile ] `thenSkip` spacing
+    return $ T_FdRedirect id n redir
 
 readRedirectList = many1 readIoRedirect
 
@@ -1599,18 +1714,26 @@ readHereString = called "here string" $ do
     spacing
     id2 <- getNextId
     word <- readNormalWord
-    return $ T_FdRedirect id "" $ T_HereString id2 word
+    return $ T_HereString id2 word
 
 readNewlineList = many1 ((linefeed <|> carriageReturn) `thenSkip` spacing)
 readLineBreak = optional readNewlineList
 
 prop_readSeparator1 = isWarning readScript "a &; b"
 prop_readSeparator2 = isOk readScript "a & b"
+prop_readSeparator3 = isWarning readScript "a &amp; b"
+prop_readSeparator4 = isWarning readScript "a &gt; file; b"
 readSeparatorOp = do
     notFollowedBy2 (void g_AND_IF <|> void readCaseSeparator)
     notFollowedBy2 (string "&>")
     f <- try (do
+                    pos <- getPosition
                     char '&'
+                    optional $ do
+                        s <- lookAhead . choice . map (try . string) $
+                            ["amp;", "gt;", "lt;"]
+                        parseProblemAt pos ErrorC 1109 "This is an unquoted HTML entity. Replace with corresponding character."
+
                     spacing
                     pos <- getPosition
                     char ';'
@@ -1878,7 +2001,7 @@ readElifPart = called "elif clause" $ do
     pos <- getPosition
     correctElif <- elif
     unless correctElif $
-        parseProblemAt pos ErrorC 1075 "Use 'elif' instead of 'else if'."
+        parseProblemAt pos ErrorC 1075 "Use 'elif' instead of 'else if' (or put 'if' on new line if nesting)."
     allspacing
     condition <- readTerm
 
@@ -2176,8 +2299,8 @@ readCompoundCommand = do
     id <- getNextId
     cmd <- choice [
         readBraceGroup,
-        readTripleParenthesis "" readArithmeticExpression readSubshell,
-        readArithmeticExpression,
+        readAmbiguous "((" readArithmeticExpression readSubshell (\pos ->
+            parseNoteAt pos WarningC 1105 "Shells disambiguate (( differently or not at all. For subshell, add spaces around ( . For ((, fix parsing errors."),
         readSubshell,
         readCondition,
         readWhileClause,
@@ -2205,7 +2328,7 @@ readCompoundListOrEmpty = do
 
 readCmdPrefix = many1 (readIoRedirect <|> readAssignmentWord)
 readCmdSuffix = many1 (readIoRedirect <|> readCmdWord)
-readModifierSuffix = many1 (readIoRedirect <|> readAssignmentWord <|> readCmdWord)
+readModifierSuffix = many1 (readIoRedirect <|> readWellFormedAssignment <|> readCmdWord)
 readTimeSuffix = do
     flags <- many readFlag
     pipeline <- readPipeline
@@ -2251,7 +2374,7 @@ readEvalSuffix = many1 (readIoRedirect <|> readCmdWord <|> evalFallback)
 
 -- Get whatever a parser would parse as a string
 readStringForParser parser = do
-    pos <- lookAhead (parser >> getPosition)
+    pos <- inSeparateContext $ lookAhead (parser >> getPosition)
     readUntil pos
   where
     readUntil endPos = anyChar `reluctantlyTill` (getPosition >>= guard . (== endPos))
@@ -2271,14 +2394,18 @@ prop_readAssignmentWord9c= isOk readAssignmentWord "foo=  #bar"
 prop_readAssignmentWord10= isWarning readAssignmentWord "foo$n=42"
 prop_readAssignmentWord11= isOk readAssignmentWord "foo=([a]=b [c] [d]= [e f )"
 prop_readAssignmentWord12= isOk readAssignmentWord "a[b <<= 3 + c]='thing'"
-readAssignmentWord = try $ do
+readAssignmentWord = readAssignmentWordExt True
+readWellFormedAssignment = readAssignmentWordExt False
+readAssignmentWordExt lenient = try $ do
     id <- getNextId
     pos <- getPosition
-    optional (char '$' >> parseNote ErrorC 1066 "Don't use $ on the left side of assignments.")
+    when lenient $
+        optional (char '$' >> parseNote ErrorC 1066 "Don't use $ on the left side of assignments.")
     variable <- readVariableName
-    optional (readNormalDollar >> parseNoteAt pos ErrorC
+    when lenient $
+        optional (readNormalDollar >> parseNoteAt pos ErrorC
                                 1067 "For indirection, use (associative) arrays or 'read \"var$n\" <<< \"value\"'")
-    index <- optionMaybe readArrayIndex
+    indices <- many readArrayIndex
     hasLeftSpace <- liftM (not . null) spacing
     pos <- getPosition
     op <- readAssignmentOp
@@ -2290,13 +2417,13 @@ readAssignmentWord = try $ do
             parseNoteAt pos WarningC 1007
                 "Remove space after = if trying to assign a value (for empty string, use var='' ... )."
         value <- readEmptyLiteral
-        return $ T_Assignment id op variable index value
+        return $ T_Assignment id op variable indices value
       else do
         when (hasLeftSpace || hasRightSpace) $
             parseNoteAt pos ErrorC 1068 "Don't put spaces around the = in assignments."
         value <- readArray <|> readNormalWord
         spacing
-        return $ T_Assignment id op variable index value
+        return $ T_Assignment id op variable indices value
   where
     readAssignmentOp = do
         pos <- getPosition
@@ -2316,12 +2443,14 @@ readAssignmentWord = try $ do
         return $ T_Literal id ""
 
 readArrayIndex = do
+    id <- getNextId
     char '['
-    optional space
-    x <- readArithmeticContents
+    pos <- getPosition
+    str <- readStringForParser readIndexSpan
     char ']'
-    return x
+    return $ T_UnparsedIndex id pos str
 
+readArray :: Monad m => SCParser m Token
 readArray = called "array assignment" $ do
     id <- getNextId
     char '('
@@ -2334,7 +2463,7 @@ readArray = called "array assignment" $ do
     readIndexed = do
         id <- getNextId
         index <- try $ do
-            x <- readArrayIndex
+            x <- many1 readArrayIndex
             char '='
             return x
         value <- readNormalWord <|> nothing
@@ -2477,12 +2606,12 @@ verifyEof = eof <|> choice [
         try (lookAhead p)
         action
 
-prop_readScript1 = isOk readScript "#!/bin/bash\necho hello world\n"
-prop_readScript2 = isWarning readScript "#!/bin/bash\r\necho hello world\n"
-prop_readScript3 = isWarning readScript "#!/bin/bash\necho hello\xA0world"
-prop_readScript4 = isWarning readScript "#!/usr/bin/perl\nfoo=("
-prop_readScript5 = isOk readScript "#!/bin/bash\n#This is an empty script\n\n"
-readScript = do
+prop_readScript1 = isOk readScriptFile "#!/bin/bash\necho hello world\n"
+prop_readScript2 = isWarning readScriptFile "#!/bin/bash\r\necho hello world\n"
+prop_readScript3 = isWarning readScriptFile "#!/bin/bash\necho hello\xA0world"
+prop_readScript4 = isWarning readScriptFile "#!/usr/bin/perl\nfoo=("
+prop_readScript5 = isOk readScriptFile "#!/bin/bash\n#This is an empty script\n\n"
+readScriptFile = do
     id <- getNextId
     pos <- getPosition
     optional $ do
@@ -2493,11 +2622,13 @@ readScript = do
     verifyShell pos (getShell sb)
     if isValidShell (getShell sb) /= Just False
       then do
+            allspacing
             annotationId <- getNextId
             annotations <- readAnnotations
             commands <- withAnnotations annotations readCompoundListOrEmpty
             verifyEof
-            return $ T_Annotation annotationId annotations $  T_Script id sb commands
+            let script = T_Annotation annotationId annotations $  T_Script id sb commands
+            reparseIndices script
         else do
             many anyChar
             return $ T_Script id sb []
@@ -2516,8 +2647,8 @@ readScript = do
     verifyShell pos s =
         case isValidShell s of
             Just True -> return ()
-            Just False -> parseProblemAt pos ErrorC 1071 "ShellCheck only supports sh/bash/ksh scripts. Sorry!"
-            Nothing -> parseProblemAt pos InfoC 1008 "This shebang was unrecognized. Note that ShellCheck only handles sh/bash/ksh."
+            Just False -> parseProblemAt pos ErrorC 1071 "ShellCheck only supports sh/bash/dash/ksh scripts. Sorry!"
+            Nothing -> parseProblemAt pos InfoC 1008 "This shebang was unrecognized. Note that ShellCheck only handles sh/bash/dash/ksh."
 
     isValidShell s =
         let good = s == "" || any (`isPrefixOf` s) goodShells
@@ -2549,6 +2680,9 @@ readScript = do
 
     readUtf8Bom = called "Byte Order Mark" $ string "\xFEFF"
 
+readScript = do
+    script <- readScriptFile
+    reparseIndices script
 
 isWarning p s = parsesCleanly p s == Just False
 isOk p s =      parsesCleanly p s == Just True
@@ -2571,13 +2705,15 @@ parseWithNotes parser = do
     state <- getState
     return (item, state)
 
-compareNotes (ParseNote pos1 level1 _ s1) (ParseNote pos2 level2 _ s2) = compare (pos1, level1) (pos2, level2)
+compareNotes (ParseNote pos1 pos1' level1 _ s1) (ParseNote pos2 pos2' level2 _ s2) = compare (pos1, pos1', level1) (pos2, pos2', level2)
 sortNotes = sortBy compareNotes
 
 
 makeErrorFor parsecError =
-    ParseNote (errorPos parsecError) ErrorC 1072 $
+    ParseNote pos pos ErrorC 1072 $
         getStringFromParsec $ errorMessages parsecError
+    where
+      pos = errorPos parsecError
 
 getStringFromParsec errors =
         case map f errors of
@@ -2630,11 +2766,46 @@ parseShell sys name contents = do
     isName (ContextName _ _) = True
     isName _ = False
     notesForContext list = zipWith ($) [first, second] $ filter isName list
-    first (ContextName pos str) = ParseNote pos ErrorC 1073 $
+    first (ContextName pos str) = ParseNote pos pos ErrorC 1073 $
         "Couldn't parse this " ++ str ++ "."
-    second (ContextName pos str) = ParseNote pos InfoC 1009 $
+    second (ContextName pos str) = ParseNote pos pos InfoC 1009 $
         "The mentioned parser error was in this " ++ str ++ "."
 
+-- Go over all T_UnparsedIndex and reparse them as either arithmetic or text
+-- depending on declare -A statements.
+reparseIndices root =
+   analyze blank blank f root
+  where
+    associative = getAssociativeArrays root
+    isAssociative s = s `elem` associative
+    f (T_Assignment id mode name indices value) = do
+        newIndices <- mapM (fixAssignmentIndex name) indices
+        newValue <- case value of
+            (T_Array id2 words) -> do
+                newWords <- mapM (fixIndexElement name) words
+                return $ T_Array id2 newWords
+            x -> return x
+        return $ T_Assignment id mode name newIndices newValue
+    f t = return t
+
+    fixIndexElement name word =
+        case word of
+            T_IndexedElement id indices value -> do
+                new <- mapM (fixAssignmentIndex name) indices
+                return $ T_IndexedElement id new value
+            otherwise -> return word
+
+    fixAssignmentIndex name word =
+        case word of
+            T_UnparsedIndex id pos src -> do
+                parsed name pos src
+            otherwise -> return word
+
+    parsed name pos src =
+        if isAssociative name
+        then subParse pos (called "associative array index" $ readIndexSpan) src
+        else subParse pos (called "arithmetic array index expression" $ optional space >> readArithmeticContents) src
+
 reattachHereDocs root map =
     doTransform f root
   where
@@ -2644,8 +2815,8 @@ reattachHereDocs root map =
     f t = t
 
 toPositionedComment :: ParseNote -> PositionedComment
-toPositionedComment (ParseNote pos severity code message) =
-    PositionedComment (posToPos pos) $ Comment severity code message
+toPositionedComment (ParseNote start end severity code message) =
+    PositionedComment (posToPos start) (posToPos end) $ Comment severity code message
 
 posToPos :: SourcePos -> Position
 posToPos sp = Position {

nextnumber

@@ -0,0 +1,10 @@
+#!/bin/bash
+# TODO: Find a less trashy way to get the next available error code
+
+shopt -s globstar
+
+for i in 1 2
+do
+  last=$(grep -hv "^prop" **/*.hs | grep -Ewo "$i[0-9]{3}" | sort -n | tail -n 1)
+  echo "Next ${i}xxx: $((last+1))"
+done

quicktest

@@ -9,6 +9,7 @@
   ,ShellCheck.Parser.runTests
   ,ShellCheck.Checker.runTests
   ,ShellCheck.Checks.Commands.runTests
+  ,ShellCheck.Checks.ShellSupport.runTests
   ,ShellCheck.AnalyzerLib.runTests
     ]' | tr -d '\n' | cabal repl 2>&1 | tee /dev/stderr)
 if [[ $var == *$'\nTrue'* ]]

shellcheck.1.md

@@ -175,6 +175,15 @@ ShellCheck uses the follow exit codes:
 + 3: ShellCheck was invoked with bad syntax (e.g. unknown flag).
 + 4: ShellCheck was invoked with bad options (e.g. unknown formatter).
 
+# LOCALE
+This version of ShellCheck is only available in English. All files are
+leniently decoded as UTF-8, with a fallback of ISO-8859-1 for invalid
+sequences. `LC_CTYPE` is respected for output, and defaults to UTF-8 for
+locales where encoding is unspecified (such as the `C` locale).
+
+Windows users seeing `commitBuffer: invalid argument (invalid character)`
+should set their terminal to use UTF-8 with `chcp 65001`.
+
 # AUTHOR
 ShellCheck is written and maintained by Vidar Holen.
 

shellcheck.hs

@@ -31,6 +31,7 @@ import qualified ShellCheck.Formatter.TTY
 import Control.Exception
 import Control.Monad
 import Control.Monad.Except
+import Data.Bits
 import Data.Char
 import Data.Functor
 import Data.Either
@@ -288,14 +289,47 @@ ioInterface options files = do
         fallback path _ = return path
 
 inputFile file = do
-    contents <-
+    handle <-
             if file == "-"
-            then getContents
-            else readFile file
+            then return stdin
+            else openBinaryFile file ReadMode
+
+    hSetBinaryMode handle True
+    contents <- decodeString <$> hGetContents handle -- closes handle
 
     seq (length contents) $
         return contents
 
+-- Decode a char8 string into a utf8 string, with fallback on
+-- ISO-8859-1. This avoids depending on additional libraries.
+decodeString = decode
+  where
+    decode [] = []
+    decode (c:rest) | isAscii c = c : decode rest
+    decode (c:rest) =
+        let num = (fromIntegral $ ord c) :: Int
+            next = case num of
+                _ | num >= 0xF8 -> Nothing
+                  | num >= 0xF0 -> construct (num .&. 0x07) 3 rest
+                  | num >= 0xE0 -> construct (num .&. 0x0F) 2 rest
+                  | num >= 0xC0 -> construct (num .&. 0x1F) 1 rest
+                  | True -> Nothing
+        in
+            case next of
+                Just (n, remainder) -> chr n : decode remainder
+                Nothing -> c : decode rest
+
+    construct x 0 rest = do
+        guard $ x <= 0x10FFFF
+        return (x, rest)
+    construct x n (c:rest) =
+        let num = (fromIntegral $ ord c) :: Int in
+            if num >= 0x80 && num <= 0xBF
+            then construct ((x `shiftL` 6) .|. (num .&. 0x3f)) (n-1) rest
+            else Nothing
+    construct _ _ _ = Nothing
+
+
 verifyFiles files =
     when (null files) $ do
         printErr "No files specified.\n"

test/shellcheck.hs

@@ -7,12 +7,14 @@ import qualified ShellCheck.Analytics
 import qualified ShellCheck.AnalyzerLib
 import qualified ShellCheck.Parser
 import qualified ShellCheck.Checks.Commands
+import qualified ShellCheck.Checks.ShellSupport
 
 main = do
     putStrLn "Running ShellCheck tests..."
     results <- sequence [
         ShellCheck.Checker.runTests,
         ShellCheck.Checks.Commands.runTests,
+        ShellCheck.Checks.ShellSupport.runTests,
         ShellCheck.Analytics.runTests,
         ShellCheck.AnalyzerLib.runTests,
         ShellCheck.Parser.runTests