Copyright | (c) 2009, 2010, 2011, 2012 Bryan O'Sullivan, (c) 2009 Duncan Coutts, (c) 2008, 2009 Tom Harper |
---|---|
License | BSD-style |
Maintainer | bos@serpentine.com |
Stability | experimental |
Portability | GHC |
Safe Haskell | Trustworthy |
Language | Haskell98 |
A time and space-efficient implementation of Unicode text. Suitable for performance critical use, both in terms of large data quantities and high speed.
Note: Read below the synopsis for important notes on the use of this module.
This module is intended to be imported qualified
, to avoid name
clashes with Prelude functions, e.g.
import qualified Data.Text as T
To use an extended and very rich family of functions for working with Unicode text (including normalization, regular expressions, non-standard encodings, text breaking, and locales), see the text-icu package.
- data Text
- pack :: String -> Text
- unpack :: Text -> String
- singleton :: Char -> Text
- empty :: Text
- cons :: Char -> Text -> Text
- snoc :: Text -> Char -> Text
- append :: Text -> Text -> Text
- uncons :: Text -> Maybe (Char, Text)
- head :: Text -> Char
- last :: Text -> Char
- tail :: Text -> Text
- init :: Text -> Text
- null :: Text -> Bool
- length :: Text -> Int
- compareLength :: Text -> Int -> Ordering
- map :: (Char -> Char) -> Text -> Text
- intercalate :: Text -> [Text] -> Text
- intersperse :: Char -> Text -> Text
- transpose :: [Text] -> [Text]
- reverse :: Text -> Text
- replace :: Text -> Text -> Text -> Text
- toCaseFold :: Text -> Text
- toLower :: Text -> Text
- toUpper :: Text -> Text
- toTitle :: Text -> Text
- justifyLeft :: Int -> Char -> Text -> Text
- justifyRight :: Int -> Char -> Text -> Text
- center :: Int -> Char -> Text -> Text
- foldl :: (a -> Char -> a) -> a -> Text -> a
- foldl' :: (a -> Char -> a) -> a -> Text -> a
- foldl1 :: (Char -> Char -> Char) -> Text -> Char
- foldl1' :: (Char -> Char -> Char) -> Text -> Char
- foldr :: (Char -> a -> a) -> a -> Text -> a
- foldr1 :: (Char -> Char -> Char) -> Text -> Char
- concat :: [Text] -> Text
- concatMap :: (Char -> Text) -> Text -> Text
- any :: (Char -> Bool) -> Text -> Bool
- all :: (Char -> Bool) -> Text -> Bool
- maximum :: Text -> Char
- minimum :: Text -> Char
- scanl :: (Char -> Char -> Char) -> Char -> Text -> Text
- scanl1 :: (Char -> Char -> Char) -> Text -> Text
- scanr :: (Char -> Char -> Char) -> Char -> Text -> Text
- scanr1 :: (Char -> Char -> Char) -> Text -> Text
- mapAccumL :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)
- mapAccumR :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)
- replicate :: Int -> Text -> Text
- unfoldr :: (a -> Maybe (Char, a)) -> a -> Text
- unfoldrN :: Int -> (a -> Maybe (Char, a)) -> a -> Text
- take :: Int -> Text -> Text
- takeEnd :: Int -> Text -> Text
- drop :: Int -> Text -> Text
- dropEnd :: Int -> Text -> Text
- takeWhile :: (Char -> Bool) -> Text -> Text
- takeWhileEnd :: (Char -> Bool) -> Text -> Text
- dropWhile :: (Char -> Bool) -> Text -> Text
- dropWhileEnd :: (Char -> Bool) -> Text -> Text
- dropAround :: (Char -> Bool) -> Text -> Text
- strip :: Text -> Text
- stripStart :: Text -> Text
- stripEnd :: Text -> Text
- splitAt :: Int -> Text -> (Text, Text)
- breakOn :: Text -> Text -> (Text, Text)
- breakOnEnd :: Text -> Text -> (Text, Text)
- break :: (Char -> Bool) -> Text -> (Text, Text)
- span :: (Char -> Bool) -> Text -> (Text, Text)
- group :: Text -> [Text]
- groupBy :: (Char -> Char -> Bool) -> Text -> [Text]
- inits :: Text -> [Text]
- tails :: Text -> [Text]
- splitOn :: Text -> Text -> [Text]
- split :: (Char -> Bool) -> Text -> [Text]
- chunksOf :: Int -> Text -> [Text]
- lines :: Text -> [Text]
- words :: Text -> [Text]
- unlines :: [Text] -> Text
- unwords :: [Text] -> Text
- isPrefixOf :: Text -> Text -> Bool
- isSuffixOf :: Text -> Text -> Bool
- isInfixOf :: Text -> Text -> Bool
- stripPrefix :: Text -> Text -> Maybe Text
- stripSuffix :: Text -> Text -> Maybe Text
- commonPrefixes :: Text -> Text -> Maybe (Text, Text, Text)
- filter :: (Char -> Bool) -> Text -> Text
- breakOnAll :: Text -> Text -> [(Text, Text)]
- find :: (Char -> Bool) -> Text -> Maybe Char
- partition :: (Char -> Bool) -> Text -> (Text, Text)
- index :: Text -> Int -> Char
- findIndex :: (Char -> Bool) -> Text -> Maybe Int
- count :: Text -> Text -> Int
- zip :: Text -> Text -> [(Char, Char)]
- zipWith :: (Char -> Char -> Char) -> Text -> Text -> Text
- copy :: Text -> Text
- unpackCString# :: Addr# -> Text
Strict vs lazy types
This package provides both strict and lazy Text
types. The
strict type is provided by the Data.Text module, while the lazy
type is provided by the Data.Text.Lazy module. Internally, the
lazy Text
type consists of a list of strict chunks.
The strict Text
type requires that an entire string fit into
memory at once. The lazy Text
type is capable of
streaming strings that are larger than memory using a small memory
footprint. In many cases, the overhead of chunked streaming makes
the lazy Text
type slower than its strict
counterpart, but this is not always the case. Sometimes, the time
complexity of a function in one module may be different from the
other, due to their differing internal structures.
Each module provides an almost identical API, with the main
difference being that the strict module uses Int
values for
lengths and counts, while the lazy module uses Int64
lengths.
Acceptable data
A Text
value is a sequence of Unicode scalar values, as defined
in
§3.9, definition D76 of the Unicode 5.2 standard.
As such, a Text
cannot contain values in the range U+D800 to
U+DFFF inclusive. Haskell implementations admit all Unicode code
points
(§3.4, definition D10)
as Char
values, including code points from this invalid range.
This means that there are some Char
values that are not valid
Unicode scalar values, and the functions in this module must handle
those cases.
Within this module, many functions construct a Text
from one or
more Char
values. Those functions will substitute Char
values
that are not valid Unicode scalar values with the replacement
character "�" (U+FFFD). Functions that perform this
inspection and replacement are documented with the phrase
"Performs replacement on invalid scalar values".
(One reason for this policy of replacement is that internally, a
Text
value is represented as packed UTF-16 data. Values in the
range U+D800 through U+DFFF are used by UTF-16 to denote surrogate
code points, and so cannot be represented. The functions replace
invalid scalar values, instead of dropping them, as a security
measure. For details, see
Unicode Technical Report 36, §3.5.)
Fusion
Most of the functions in this module are subject to fusion,
meaning that a pipeline of such functions will usually allocate at
most one Text
value.
As an example, consider the following pipeline:
import Data.Text as T import Data.Text.Encoding as E import Data.ByteString (ByteString) countChars :: ByteString -> Int countChars = T.length . T.toUpper . E.decodeUtf8
From the type signatures involved, this looks like it should
allocate one ByteString
value, and two Text
values. However, when a module is compiled with optimisation
enabled under GHC, the two intermediate Text
values will be
optimised away, and the function will be compiled down to a single
loop over the source ByteString
.
Functions that can be fused by the compiler are documented with the phrase "Subject to fusion".
Types
Creation and elimination
singleton :: Char -> Text Source
O(1) Convert a character into a Text. Subject to fusion. Performs replacement on invalid scalar values.
Basic interface
cons :: Char -> Text -> Text infixr 5 Source
O(n) Adds a character to the front of a Text
. This function
is more costly than its List
counterpart because it requires
copying a new array. Subject to fusion. Performs replacement on
invalid scalar values.
snoc :: Text -> Char -> Text Source
O(n) Adds a character to the end of a Text
. This copies the
entire array in the process, unless fused. Subject to fusion.
Performs replacement on invalid scalar values.
O(1) Returns the first character of a Text
, which must be
non-empty. Subject to fusion.
O(1) Returns the last character of a Text
, which must be
non-empty. Subject to fusion.
O(1) Returns all characters after the head of a Text
, which
must be non-empty. Subject to fusion.
O(1) Returns all but the last character of a Text
, which must
be non-empty. Subject to fusion.
compareLength :: Text -> Int -> Ordering Source
Transformations
intercalate :: Text -> [Text] -> Text Source
O(n) The intercalate
function takes a Text
and a list of
Text
s and concatenates the list after interspersing the first
argument between each element of the list.
intersperse :: Char -> Text -> Text Source
O(n) The intersperse
function takes a character and places it
between the characters of a Text
. Subject to fusion. Performs
replacement on invalid scalar values.
:: Text |
|
-> Text |
|
-> Text |
|
-> Text |
O(m+n) Replace every non-overlapping occurrence of needle
in
haystack
with replacement
.
This function behaves as though it was defined as follows:
replace needle replacement haystack =intercalate
replacement (splitOn
needle haystack)
As this suggests, each occurrence is replaced exactly once. So if
needle
occurs in replacement
, that occurrence will not itself
be replaced recursively:
replace "oo" "foo" "oo" == "foo"
In cases where several instances of needle
overlap, only the
first one will be replaced:
replace "ofo" "bar" "ofofo" == "barfo"
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
Case conversion
When case converting Text
values, do not use combinators like
map toUpper
to case convert each character of a string
individually, as this gives incorrect results according to the
rules of some writing systems. The whole-string case conversion
functions from this module, such as toUpper
, obey the correct
case conversion rules. As a result, these functions may map one
input character to two or three output characters. For examples,
see the documentation of each function.
Note: In some languages, case conversion is a locale- and context-dependent operation. The case conversion functions in this module are not locale sensitive. Programs that require locale sensitivity should use appropriate versions of the case mapping functions from the text-icu package.
toCaseFold :: Text -> Text Source
O(n) Convert a string to folded case. Subject to fusion.
This function is mainly useful for performing caseless (also known as case insensitive) string comparisons.
A string x
is a caseless match for a string y
if and only if:
toCaseFold x == toCaseFold y
The result string may be longer than the input string, and may
differ from applying toLower
to the input string. For instance,
the Armenian small ligature "ﬓ" (men now, U+FB13) is case
folded to the sequence "մ" (men, U+0574) followed by
"ն" (now, U+0576), while the Greek "µ" (micro sign,
U+00B5) is case folded to "μ" (small letter mu, U+03BC)
instead of itself.
toLower :: Text -> Text Source
O(n) Convert a string to lower case, using simple case conversion. Subject to fusion.
The result string may be longer than the input string. For instance, "İ" (Latin capital letter I with dot above, U+0130) maps to the sequence "i" (Latin small letter i, U+0069) followed by " ̇" (combining dot above, U+0307).
toUpper :: Text -> Text Source
O(n) Convert a string to upper case, using simple case conversion. Subject to fusion.
The result string may be longer than the input string. For instance, the German "ß" (eszett, U+00DF) maps to the two-letter sequence "SS".
toTitle :: Text -> Text Source
O(n) Convert a string to title case, using simple case conversion. Subject to fusion.
The first letter of the input is converted to title case, as is every subsequent letter that immediately follows a non-letter. Every letter that immediately follows another letter is converted to lower case.
The result string may be longer than the input string. For example, the Latin small ligature fl (U+FB02) is converted to the sequence Latin capital letter F (U+0046) followed by Latin small letter l (U+006C).
Note: this function does not take language or culture specific rules into account. For instance, in English, different style guides disagree on whether the book name "The Hill of the Red Fox" is correctly title cased—but this function will capitalize every word.
Justification
justifyLeft :: Int -> Char -> Text -> Text Source
O(n) Left-justify a string to the given length, using the specified fill character on the right. Subject to fusion. Performs replacement on invalid scalar values.
Examples:
justifyLeft 7 'x' "foo" == "fooxxxx" justifyLeft 3 'x' "foobar" == "foobar"
justifyRight :: Int -> Char -> Text -> Text Source
O(n) Right-justify a string to the given length, using the specified fill character on the left. Performs replacement on invalid scalar values.
Examples:
justifyRight 7 'x' "bar" == "xxxxbar" justifyRight 3 'x' "foobar" == "foobar"
center :: Int -> Char -> Text -> Text Source
O(n) Center a string to the given length, using the specified fill character on either side. Performs replacement on invalid scalar values.
Examples:
center 8 'x' "HS" = "xxxHSxxx"
Folds
foldl' :: (a -> Char -> a) -> a -> Text -> a Source
O(n) A strict version of foldl
. Subject to fusion.
foldl1' :: (Char -> Char -> Char) -> Text -> Char Source
O(n) A strict version of foldl1
. Subject to fusion.
Special folds
Construction
Scans
Accumulating maps
mapAccumR :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text) Source
The mapAccumR
function behaves like a combination of map
and
a strict foldr
; it applies a function to each element of a
Text
, passing an accumulating parameter from right to left, and
returning a final value of this accumulator together with the new
Text
.
Performs replacement on invalid scalar values.
Generation and unfolding
unfoldr :: (a -> Maybe (Char, a)) -> a -> Text Source
O(n), where n
is the length of the result. The unfoldr
function is analogous to the List unfoldr
. unfoldr
builds a
Text
from a seed value. The function takes the element and
returns Nothing
if it is done producing the Text
, otherwise
Just
(a,b)
. In this case, a
is the next Char
in the
string, and b
is the seed value for further production. Subject
to fusion. Performs replacement on invalid scalar values.
unfoldrN :: Int -> (a -> Maybe (Char, a)) -> a -> Text Source
O(n) Like unfoldr
, unfoldrN
builds a Text
from a seed
value. However, the length of the result should be limited by the
first argument to unfoldrN
. This function is more efficient than
unfoldr
when the maximum length of the result is known and
correct, otherwise its performance is similar to unfoldr
. Subject
to fusion. Performs replacement on invalid scalar values.
Substrings
Breaking strings
takeEnd :: Int -> Text -> Text Source
O(n) takeEnd
n
t
returns the suffix remaining after
taking n
characters from the end of t
.
Examples:
takeEnd 3 "foobar" == "bar"
dropEnd :: Int -> Text -> Text Source
O(n) dropEnd
n
t
returns the prefix remaining after
dropping n
characters from the end of t
.
Examples:
dropEnd 3 "foobar" == "foo"
takeWhileEnd :: (Char -> Bool) -> Text -> Text Source
O(n) takeWhileEnd
, applied to a predicate p
and a Text
,
returns the longest suffix (possibly empty) of elements that
satisfy p
. Subject to fusion.
Examples:
takeWhileEnd (=='o') "foo" == "oo"
dropWhileEnd :: (Char -> Bool) -> Text -> Text Source
O(n) dropWhileEnd
p
t
returns the prefix remaining after
dropping characters that satisfy the predicate p
from the end of
t
. Subject to fusion.
Examples:
dropWhileEnd (=='.') "foo..." == "foo"
dropAround :: (Char -> Bool) -> Text -> Text Source
O(n) dropAround
p
t
returns the substring remaining after
dropping characters that satisfy the predicate p
from both the
beginning and end of t
. Subject to fusion.
O(n) Remove leading and trailing white space from a string. Equivalent to:
dropAround isSpace
stripStart :: Text -> Text Source
O(n) Remove leading white space from a string. Equivalent to:
dropWhile isSpace
stripEnd :: Text -> Text Source
O(n) Remove trailing white space from a string. Equivalent to:
dropWhileEnd isSpace
breakOn :: Text -> Text -> (Text, Text) Source
O(n+m) Find the first instance of needle
(which must be
non-null
) in haystack
. The first element of the returned tuple
is the prefix of haystack
before needle
is matched. The second
is the remainder of haystack
, starting with the match.
Examples:
breakOn "::" "a::b::c" ==> ("a", "::b::c") breakOn "/" "foobar" ==> ("foobar", "")
Laws:
append prefix match == haystack where (prefix, match) = breakOn needle haystack
If you need to break a string by a substring repeatedly (e.g. you
want to break on every instance of a substring), use breakOnAll
instead, as it has lower startup overhead.
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
breakOnEnd :: Text -> Text -> (Text, Text) Source
O(n+m) Similar to breakOn
, but searches from the end of the
string.
The first element of the returned tuple is the prefix of haystack
up to and including the last match of needle
. The second is the
remainder of haystack
, following the match.
breakOnEnd "::" "a::b::c" ==> ("a::b::", "c")
span :: (Char -> Bool) -> Text -> (Text, Text) Source
O(n) span
, applied to a predicate p
and text t
, returns
a pair whose first element is the longest prefix (possibly empty)
of t
of elements that satisfy p
, and whose second is the
remainder of the list.
groupBy :: (Char -> Char -> Bool) -> Text -> [Text] Source
O(n) Group characters in a string according to a predicate.
Breaking into many substrings
Splitting functions in this library do not perform character-wise
copies to create substrings; they just construct new Text
s that
are slices of the original.
O(m+n) Break a Text
into pieces separated by the first Text
argument (which cannot be empty), consuming the delimiter. An empty
delimiter is invalid, and will cause an error to be raised.
Examples:
splitOn "\r\n" "a\r\nb\r\nd\r\ne" == ["a","b","d","e"] splitOn "aaa" "aaaXaaaXaaaXaaa" == ["","X","X","X",""] splitOn "x" "x" == ["",""]
and
intercalate s . splitOn s == id splitOn (singleton c) == split (==c)
(Note: the string s
to split on above cannot be empty.)
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
split :: (Char -> Bool) -> Text -> [Text] Source
O(n) Splits a Text
into components delimited by separators,
where the predicate returns True for a separator element. The
resulting components do not contain the separators. Two adjacent
separators result in an empty component in the output. eg.
split (=='a') "aabbaca" == ["","","bb","c",""] split (=='a') "" == [""]
chunksOf :: Int -> Text -> [Text] Source
O(n) Splits a Text
into components of length k
. The last
element may be shorter than the other chunks, depending on the
length of the input. Examples:
chunksOf 3 "foobarbaz" == ["foo","bar","baz"] chunksOf 4 "haskell.org" == ["hask","ell.","org"]
Breaking into lines and words
Predicates
isPrefixOf :: Text -> Text -> Bool Source
O(n) The isPrefixOf
function takes two Text
s and returns
True
iff the first is a prefix of the second. Subject to fusion.
isSuffixOf :: Text -> Text -> Bool Source
O(n) The isSuffixOf
function takes two Text
s and returns
True
iff the first is a suffix of the second.
View patterns
stripPrefix :: Text -> Text -> Maybe Text Source
O(n) Return the suffix of the second string if its prefix matches the entire first string.
Examples:
stripPrefix "foo" "foobar" == Just "bar" stripPrefix "" "baz" == Just "baz" stripPrefix "foo" "quux" == Nothing
This is particularly useful with the ViewPatterns
extension to
GHC, as follows:
{-# LANGUAGE ViewPatterns #-} import Data.Text as T fnordLength :: Text -> Int fnordLength (stripPrefix "fnord" -> Just suf) = T.length suf fnordLength _ = -1
stripSuffix :: Text -> Text -> Maybe Text Source
O(n) Return the prefix of the second string if its suffix matches the entire first string.
Examples:
stripSuffix "bar" "foobar" == Just "foo" stripSuffix "" "baz" == Just "baz" stripSuffix "foo" "quux" == Nothing
This is particularly useful with the ViewPatterns
extension to
GHC, as follows:
{-# LANGUAGE ViewPatterns #-} import Data.Text as T quuxLength :: Text -> Int quuxLength (stripSuffix "quux" -> Just pre) = T.length pre quuxLength _ = -1
commonPrefixes :: Text -> Text -> Maybe (Text, Text, Text) Source
O(n) Find the longest non-empty common prefix of two strings and return it, along with the suffixes of each string at which they no longer match.
If the strings do not have a common prefix or either one is empty,
this function returns Nothing
.
Examples:
commonPrefixes "foobar" "fooquux" == Just ("foo","bar","quux") commonPrefixes "veeble" "fetzer" == Nothing commonPrefixes "" "baz" == Nothing
Searching
O(n+m) Find all non-overlapping instances of needle
in
haystack
. Each element of the returned list consists of a pair:
- The entire string prior to the kth match (i.e. the prefix)
- The kth match, followed by the remainder of the string
Examples:
breakOnAll "::" "" ==> [] breakOnAll "/" "a/b/c/" ==> [("a", "/b/c/"), ("a/b", "/c/"), ("a/b/c", "/")]
In (unlikely) bad cases, this function's time complexity degrades towards O(n*m).
The needle
parameter may not be empty.
Indexing
If you think of a Text
value as an array of Char
values (which
it is not), you run the risk of writing inefficient code.
An idiom that is common in some languages is to find the numeric
offset of a character or substring, then use that number to split
or trim the searched string. With a Text
value, this approach
would require two O(n) operations: one to perform the search, and
one to operate from wherever the search ended.
For example, suppose you have a string that you want to split on
the substring "::"
, such as "foo::bar::quux"
. Instead of
searching for the index of "::"
and taking the substrings
before and after that index, you would instead use breakOnAll "::"
.
Zipping
Low level operations
O(n) Make a distinct copy of the given string, sharing no storage with the original string.
As an example, suppose you read a large string, of which you need
only a small portion. If you do not use copy
, the entire original
array will be kept alive in memory by the smaller string. Making a
copy "breaks the link" to the original array, allowing it to be
garbage collected if there are no other live references to it.
unpackCString# :: Addr# -> Text Source
O(n) Convert a literal string into a Text
. Subject to
fusion.
This is exposed solely for people writing GHC rewrite rules.