Crandas Changelog - Roseman Labs (2024)

Crandas¶

• Fix a bug, where reducing the bounds (e.g., using astype) breaksthe server when it tries to switch to a smaller internalrepresentation.

• Allow fixedpoint ctype specification to define min and max as floats.For example:

  table = cd.DataFrame({"vals": [0.75, 1.25, 2.25]}, ctype={"vals": "fp[min=0.5,max=2.5]"})

• Add cd.stats.contingency.crosstab for computing contingencytables

• Add crandas.CSeries.as_series to obtain a ReturnValue for theseries

• Add crandas.stats.chi2_contingency for performing chisquare teston contingency table

• Add crandas.stats.contingency.expected_freq for computingexpected frequences

• Fix crandas.stats.ttest_ind() to work with unequal variances forlarger data

• Fix crandas.stats.chisquare to provide better error message wheninput contains zeros

• Added the crandas.compat, crandas.crypto andcrandas.stats packages in pyproject.toml via automatic packagediscovery.

Crandas¶

• Fix a bug in fixed-point multiplication; in certain cases this bugcaused incorrect fixed-point multiplication results.

• Fix an issue related to the printing of the min and max value of afixed-point ctype

• Fix bug where, in some cases, a schema mismatch between design andproduction would cause an exception rather than a user-friendly errormessage

Crandas¶

• Major performance improvements for comparisons and related operationslike sort_values and groupby. The performance of theseoperations is improved by around a factor 10.

• Fix bugs in substitute in the following edge-cases:

  • Fix crash when the output_size parameter is set too high.

  • Fix potential invalid substitutions when Unicode substitutions areapplied to ASCII strings.

• Add support for fixed-point arrays, including array functionsinner and vsum.

• Add support for cd.cut on fixed-point values, bins and labels.For example:

  table = cd.DataFrame({"vals":[1.1, 2.1, 2.2, 3.3]}, ctype={"vals":"fp[precision=30]"})table = table.assign(cut=cd.cut(table["vals"], [1.5,2.5], labels=[0.1,0.2,0.3], add_inf=True))

• Fix bug which prevented uploading bytes columns with more than 100000 values.

• Add support for Base64 encoding and decoding:bytes_col.b64encode() and string_col.b64decode():

  table = cd.DataFrame({"base64": ["TWFu", "bGlnaHQgd29yaw=="]}, auto_bounds=True)table = table.assign(bytes=table["base64"].b64decode()) # table["bytes"] set to [b"Man", b"light work"]table["bytes"].b64encode() # equal to table["base64"]

• Add support for performing a groupby on more than 100 000 uniquevalues.

• Add support for computing the square root cdf["col"].sqrt()

• Add module crandas.stats for performing a number of statisticaltests

• override_version_check can now be supplied to the Sessionconstructor, socd.connect("connection-file", override_version_check=False) nowworks. Additionally, the CRANDAS_OVERRIDE_VERSION_CHECKenvironment variable was moved to the cd.config framework.

• Support mode and other query arguments for CSeries.sum(),e.g.so that the following now works without opening values

  table = cd.DataFrame({"values": [1, 2, 3]})x = (table["values"] ** 2).sum(mode="regular")# x is not opened

• Improve support for using multiple sessions at the same time with thesession query argument and the use of a session as a contexthandler

  BREAKING: A check is introduced to ensure that objects fromdifferent sessions are not mixed. (Previously, if multiple sessionsto the same endpoint were used, it was possible to mix objects.)

Crandas¶

• Preserve len(df) of pandas DataFrames without columns

• Support for the concatenation of strings. For example:

  table = cd.DataFrame({"first_name": ["John", "Jan"], "last_name": ["Doe", "Jansen"]}, auto_bounds=True)full_names = table["first_name"] + " " + table["last_name"]full_names.open()

• Add support for upper(), similar to the existing lower(). Inaddition, it is now also possible to only change the case of specificindices:

  table = cd.DataFrame({"name": ["john", "Jansen"]}, auto_bounds=True)table["name"].upper([0]).open() # Returns ["John", "Jansen"]table["name"].upper([1, 3, 5]).open() # Returns ["jOhN", "JAnSeN"]

• crandas.tool.check_connection now cleans up tables after runninga dummy computation

• Perform parquet uploads through read_parquet

• Improved implementation of cd.merge:

  • Add support for right and one-to-many joins. Now, any combinationof left/right/inner/outer one-to-one/one-to-many/many-to-one joinsis supported.

  • Improved support for nullable columns. When performing a non-innerjoin, now nullable columns are output (previously, a non-nullablecolumn with the value zero was returned in some cases)

  • Deal with key columns with different colum names consistently withpandas, e.g., when joining with left_on="a" andright_on="b", return two columns with the left and right keyvalues, respectively

  • Add support for suffixes

  • Improved progress reporting for one-to-many joins

  • Allow trivial grouping-based join where the right table only haskey columns

  BREAKING: The signature of the cd.merge function has notchanged, but, because of the above changes, the resulting table mayhave a different set of columns and/or columns of different typesthan in previous versions. Moreover, the underlying VDL API commandhas been renamed and changed internally so that existingauthorizations do not apply to the new merge. The old merge isavailable via crandas as cd.compat.merge_v1.

• Fix bug in bytes column when using non-ASCII characters. Opening suchvalues could give incorrect results.

• Add support for (fixed-point) division (/). For example:

  table = cd.DataFrame({"num": [1.2, 3.4, 5.6], "denom": [2.1, 4.3, 5.4]}, auto_bounds=True)table = table.assign(div = lambda x: x.num / x.denom)table = table.assign(rec_num = lambda x: 1 / x.num) # computing the reciprocal of numtable.open()

• Add support for strip() on string columns, which will remove theleading and trailing spaces.

• Add support for floor_division (//).

• Add support for RIPEMD-160:

  import crandas.crypto.hash as hashtab = cd.DataFrame({"a": [b"Test 1", b"Test 2"]}, auto_bounds=True)h = hash.RIPEMD_160h.digest(tab["a"]).open()# HMAC is also supportedtab_key = cd.DataFrame({"key": [bytes.fromhex("0123456789abcdef0123456789abcdef01234567")]}, auto_bounds=True)hmac = hash.HMAC_RIPEMD_160(tab_key["key"].as_value())hmac.digest(tab["a"]).open()

• Add support for AES encryption:

  import crandas.crypto.cipher as ciphertable = cd.DataFrame({"a": [bytes.fromhex("00112233445566778899aabbccddeeff")] * 2}, ctype={"a": "bytes[16]"})tab_key = cd.DataFrame({"key": [bytes.fromhex("000102030405060708090a0b0c0d0e0f")]}, ctype={"key": "bytes[16]"})aes_128 = cipher.AES_128(tab_key["key"].as_value())aes_128.encrypt(table["a"]).open()

  See Also

  “trend_extrapolation” package — antiCPy 0.0.9 documentation

• Add support for len and slicing on bytes columns:bytes_col.len() and bytes_col[16:32]

• Add support for conversion to and from lowercase hex strings:string_col.hex_to_bytes() and bytes_col.bytes_to_hex()

• Add support for encoding ASCII strings as bytes:ascii_col.encode()

• Add bitwise operations on bytes columns: AND (&), OR (|), XOR(^), NEGATE (~)

• Add support for string substitution:

  table = cd.DataFrame({"a": ["PÆR", "á"]}, auto_bounds=True)table["a"].substitute({"a": ["á", "à", "ä"], "AE": ["Æ"]}, output_size=4).open()

• Add support for filtering characters:

  table = cd.DataFrame({"a": ["Test string", "More"]}, auto_bounds=True)table["a"].filter_chars(["a", "e", "i", "o", "u"]).open()

• Add support for reading stored analyst, approver, and server keys inPEM format

• Fix bug where uploading a series with only NULL values would give anerror

• Fix bug where repr(cdf), str(cdf) would not deal correctlywith zero-row dataframes

• We move auto_bounds from a Session property to be aconfiguration variable (usingcrandas.config.settings.auto_bounds). Having this variable set toTrue suppresses data-derived column bound warnings by default. Eachsession object now has a deprecated auto_bounds property thatgets/sets the configuration variable.

  BREAKING: This breaks the possibility of a user having twoconcurrent sessions with different auto_bounds values set.

• Allow to provide pd.read_csv arguments (e.g., delimiter) asarguments to cd.read_csv

• Warn user when calling crandas in a conditional context (e.g., anif statement) during script recording. See documentation of thecrandas.check_recording module for details.

• Warn users to specify a validate argument when using mergeduring script recording. See documentation of thecrandas.check_recording module for details.

• Allow to specify ctype as argument to cd.Series

• Expose ctype and schema of a column as properties of theclasses Col (cdf.columns.cols[ix]) and CSeriesColRef(cdf["col"]); and of a CDataFrame

• Add function CDataFrame.astype() that converts the type of aindividual columns (via ctype parameter) or the full CDataFrame(via schema parameter)

• Add schema parameters to upload_pandas_dataframe,read_csv, DataFrame, read_parquet functions. Forctype parameter, warn if the corresponding column does not exist

• Add functions pandas_dataframe_schema and read_csv_schemathat return the schema corresponding to a DataFrame or CSV file

• A server-side schema check for get_table is introduced. Whenget_table is used in a script, the schema of the resulting table isstored in the recorded script. When the script is used, a server-sidecheck for adherence to the schema is performed.

  BREAKING: using get_table in a script where the tables do notmatch between recording and using the script, now produces an error;see documentation of get_table for details

• Add tilde expansion to cd.base.Session.connect()

• Improved error messages for: using get_table on a non-dummyhandle in script recording; invalid arguments to cut,e.g.non-integer bins or labels; sending unauthorized queries whereauthorization is needed; invalid how argument to merge; useof None-like values in functions (e.g., x.if_else(y, None));use of unknown ctypes (e.g., ctype={"a": "str"}); uploadingfixed-point columns where integers may be intended (e.g., uploadingpd.Series([1, 2, None]))

• Fix bug where the use of a value placeholder (e.g.,cdf.assign(b=lambda x: x.a + cd.placeholders.Any(1))) would inmany cases not work

Crandas¶

• We update the pyformlang dependency to fix bugs in character ranges

Crandas¶

• We give better errors when receive unexpected responses from theserver.

• We fix a performance regression of the groupby operation, when it isperformed on a single F64 column. It is now again as fast as inversion 1.9.

Crandas¶

• Expanded support for fixed-point columns:

  • Fixed point columns now support larger range and precision (96bits).

  • Fixed point columns now support various statistical functions(min(),max(),sum(),sum_squares(),mean(), var()).

  • Support for arithmetic operations between two fixed point columns,and between fixed-point and integer columns is added. (NB: we donot yet support division; this will be added in a later release.)

  • Support for concatenation of integer and fixed point columns(resulting in a fixed-point column) is added.

  • Support for join and filtering on fixed point columns is added.

  • Parsing of floats on column operations used in operations asfilters or assign is supported.

• The new dropna function removes rows with any missing values froma CDataFrame.

• The new save can be used to save an object such as a CDataFrame.If persistence is enabled on the server, this means that the objectis kept across server restarts. The save command may also be usedto attach a name to a computed table,e.g.table.save(name="my_table").

• The connection file and Session now both have an optionalapi_token property. This is sent to the server and may be usedfor authentication purposes.

• The functions obj.remove() and cd.remove_objects() have beenchanged to provide more information in case non-existent object(s)are removed.

• Support for division is added.

  BREAKING: when removing multiple objects usingcd.remove_objects(lst), the new behavior is to try to remove allobjects even if errors are encountered. The old behavior was to aborton the first error. See the documentation for details.

Crandas¶

No changes.

Crandas¶

• The Session object now has two settings modes, depending onwhether a VDL connection file is used (recommended method), orwhether the endpoint, certificate, and server public keys arespecified manually (legacy method). These are reflected in thesettings_mode attribute of the Session object.

  When endpoint is set by the user, the Session is set tolegacy mode; otherwise, the connection file method is assumed. Whenthe user does not configure anything, the default is to load thedefault.vdlconn file, residing in the configuration folder(default: ~/.config/crandas, overridable by the CRANDAS_HOMEenvironment variable). The name default.vdlconn can be overridenthrough the default_connection_file variable. If that file is notpresent, scan the configuration folder for files with the extension.vdlconn. If there is a single file, use that. If there aremultiple, raise an error.

  analyst_key is now a read-write property that returns the naclSigningKey, and can be set to either a SigningKey, a filename, apath, or None. When set to None, the default key will be loaded. Boththe default key file, and the default relative path, depend on thesettings mode. For connection file mode, it is analyst.sk and thecurrent working directory in case of a path (Path, or a string thatincludes a slash “/”); in case of a filename (string that does notinclude a slash), it is assumed to reside in the configurationfolder; for legacy mode it is clientsign.sk and the base_path (tomaintain backwards compatibility).

• Besides the Session object, which is used to configure theconnection to the VDL, we introduceDynaconf for user configuration forsettings that are not directly related to the connection. The newmethod provides an easy way for the user to set variables, eitherusing code, using environment variables, or using a settings file(default: settings.toml in the same configuration folder referredto above).

• We make displaying progress bars configurable using theshow_progress_bar and show_progress_bar_after (for the delayin seconds) variables.

• To make the configuration folder and display the folder in the user’sfile browser, the user can now call python -m crandas config.

• We support the Any placeholder for get_table

• We support stepless mode in scripts, that can be manually enabledto remove script_step numbers from certain queries. This can beuseful together with the Any placeholder, to have queries thatcan be executed a variable number of times.

• Add a map_dummy_handles override in call to get_table

• In CDataFrame.assign, we now support the use of colum names thatcorrespond to VDL query arguments (e.g.“name”, “bitlength”).

  BREAKING: existing scripts that use these VDL query argumentswill now give an error message explaining how these arguments shouldbe specified. Existing authorizations are not affected.

• Add support for the following operators in regular expressions:

  • {n}: match exactly n times

  • {min,}: match at least min times

  • {,max}: match at most max times

  • {min,max}: match at least min and at most max times

• Support was added to disable HTTP Keep-Alive in connections to theVDL server. This can help solve connection stability issues.Keep-Alive can be disabled in the connection file by settingkeepalive = false. The setting can be overriden by the user byusing the keepalive parameter of crandas.connect.

• Add sort_values function to a CDataFrame, which sorts thedataframe according to a column. Example:

  cdf = cd.DataFrame({"a": [3, 1, 4, 5, 2], "b": [1, 2, 3, 4, 5]}, auto_bounds=True)cdf = cdf.sort_values("a")

  Currently, sorting on strings is not supported.

• Add support for groupby on multiple columns and on all non-nullablecolumn types.

  For example, this is now possible:

  cdf = cd.DataFrame({"a": ["foo", "bar", "foo", "bar"], "b": [1, 1, 1, 2]}, auto_bounds=True)tab = cdf.groupby(["a", "b"]).as_table()sorted(zip(tab["a"].open(), tab["b"].open()))

  The parameter name of the groupby is renamed from col to colsto reflect these changes. Currently, a maximum of around 100 000unique values are supported. Above that, the groupby will fail andgive an error message. Note that this is the number of uniquevalues. The number of rows can be significantly higher as long asthere are less than 100 000 different values in the groupbycolumn(s). Furthermore, a consequence of the new implementation isthat the output is not order-stable anymore but random.

• Add k-nearest neighbors functionality. This allows the target valueof a new data point to be predicted based on the existing data usingits k nearest neighbors. Example:

  import crandas as cdfrom crandas.crlearn.neighbors import KNeighborsRegressorX_train = cd.DataFrame({"input": [0, 1, 2, 3]}, auto_bounds=True)y_train = cd.DataFrame({"output": [0, 0, 1, 1]}, auto_bounds=True)X_test = cd.DataFrame({"input": [1]}, auto_bounds=True)neigh = KNeighborsRegressor(n_neighbors=3)neigh.fit(X_train, y_train)neigh.predict_value(X_test)

  For more information, seecrandas.crlearn.neighbors.KNeighborsRegressor.

• Add a new aggregator crandas.groupby.any that takes any valuefrom the set of values and is faster thancrandas.groupby.max/crandas.groupby.min

• In the HTTP connection to the VDL server, use retries for certainHTTP requests to improve robustness

• Add created property to dataframes and other objects indicatingthe date and time when they were uploaded or computed

• Handle cancellation of a query by raising aQueryInterruptedError. This replaces the previous behaviour ofreturning None and printing “Computation cancelled”. In ipython,the “Computation cancelled” message is still shown.

• In the progress bar for long-running computations, show “no estimateavailable yet” as long as progress is at 0% (instead of a morecryptic notation).

• Add functionality to list uploads to the VDL. For more information,see: crandas.stateobject.list_uploads andcrandas.stateobject.get_upload_handles.

Crandas fixes¶

• crandas.get_table() now ensures connect() is called first

• Fix upload and decoding of positive numbers of 64 bits In Crandas,trying to upload and download numbers of in the rangeR = [2^{63}, 2^{64} -1] would previously fail. We fix this issueby mimicking pandas behavior. That is, a number in the range R isreturned as an np.uint64. Secondly, w.r.t. uploading,np.uint64, np.uint32, and np.uint16 are now recognized asintegers.

New features¶

• Support for columns with bigger (96 bit) integers

  Just like in the previous version, integers have the ctype int.When specifying the ctype, minimum and maximum bounds for the valuescan be supplied using the min and max parameters,e.g.int[min=0, max=1000]. Bounds (strictly) between -2^95 and2^95 are now supported.

  For example, to upload a column "col": [1, 2, 3, 4] as an intuse the following ctype spec:

  table = cd.DataFrame({"col":[1, 2, 3, 4]}, ctype={"col": "int[min=1,max=4]"})

  as before.

  To force usage of a particular modulus the integer ctype accepts thekeyword argument modulus, which can be set to either of themoduli that are hardcoded in crandas.moduli. For example, toforce usage of large integers one can run:

  from crandas.moduli import modulitable = cd.DataFrame({"col":[1, 2, 3, 4]}, ctype={"col": f"int[min=1,max=4,modulus={moduli[128]}]"})

  Notes:

  • crandas will automatically switch toint[modulus={moduli[128]}] if the (derived) bounds do not fitin an int32.

  • crandas will throw an error if the bounds do not fit in anint96.

  We refer to 32-bit integer columns as F64, and 96-bit integer columnsas F128, because they are internally represented as 64 and 128 bitsnumbers, respectively, since we account for a necessary securitymargin.

  Supported features for large integers:

  • Basic binary arithmetic (+, -, *, ==, <, >, <=, >=) betweenany two integer columns

  • Groupby and filter on large integers

  • Unary functions on large integer columns, such asmean(), var(), sum(), ...

  • if_else where the 3 arguments guard, ifval,elseval may be any integer column

  • Conversion from 32-bit integer columns to large integer columnsvia astype and vice versa

  • Vertical concatenation of integer columns based on differentmoduli

  • Performing a join on columns based on different moduli

  Current limitations:

  • We do not yet support string conversion to large integers

  • json_to_val only allows integers up to int32 yet

  • IntegerList is only defined over F64 yet

  Changes:

  • base.py: deprecated session.modulus

  • crandas.py: class Col and ReturnValue present also themodulus

  • ctypes.py:

    • added support to encode/decode integers of 128 bits

    • made ctype class decoding modulus dependent

  • input.py: mask and unmask are now dependent on themodulus

  • placeholders.py: class Masker now also contains a modulus

  • NEW FILE moduli.py: containing the default moduli for F64 aswell as F128.

• Searching strings and regular expressions

  To search a string column for a particular substring, use theCSeries.contains function:

  table = cd.DataFrame({"col": ["this", "is", "a", "text", "column"]})only_is_rows = table["col"].contains("is")table[only_is_rows].open()

  Regular expressions are also supported, using the newCSeries.fullmatch function:

  import crandas.retable = cd.DataFrame({"col": ["this", "is", "a", "text", "column"]})starts_with_t = table["col"].fullmatch(cd.re.Re("t.*"))table[starts_with_t].open()

  Regular expressions support the following operations:

  • |: union

  • *: Kleene star (zero or or more)

  • +: one or more

  • ?: zero or one

  • .: any character (note that this also matches non-printablecharacters)

  • (, ): regexp grouping

  • [...]: set of characters (including character ranges, e.g.,[A-Za-z])

  • \\d: digits (equivalent to [0-9])

  • \\s: whitespace (equivalent to [\\\\ \\t\\n\\r\\f\\v])

  • \\w: alphanumeric and underscore (equivalent to[a-zA-Z0-9_])

  • (?1), (?2), …: substring (given as additional argument toCSeries.fullmatch())

  Regular expressions are represented by the class crandas.re.Re.It uses pyformlang’s functionality under the hood.

• Efficient text operations for ASCII strings

  The varchar ctype now has an ASCII mode for increased efficiencywith strings that do only contain ASCII characters (no “special”characters; all codepoints <= 127). Before this change, we onlysupported general Unicode strings. Certain operations (in particular,comparison, searching, and regular expression matching), are moreefficient for ASCII strings.

  By default, crandas autodetects whether or not the more efficientASCII mode can be used. This information (whether or not ASCII modeis used) becomes part of the public metadata of the column, andcrandas will give a ColumnBoundDerivedWarning to indicate thatthe column metadata is derived from the data in the column, unlessauto_bounds is set to True.

  Instead of auto-detection, it is also possible to explicitly specifythe ctype varchar[ascii] or varchar[unicode], e.g.:

  import crandas as cd# ASCII autodetected: efficient operations available; warning givencdf = cd.DataFrame({"a": ["string"]})# Unicode autodetected: efficient operations not available; warning givencdf = cd.DataFrame({"a": ["stri\U0001F600ng"]})# ASCII annotated; efficient operations available; no warning givencdf = cd.DataFrame({"a": ["string"]}, ctype={"a": "varchar[ascii]"})# Unicode annotated; efficient operations not available; no warning givencdf = cd.DataFrame({"a": ["string"]}, ctype={"a": "varchar[unicode]"})

• Running computations can now be cancelled

  Locally aborting a computation (e.g.Ctrl+C) will now cause it to becancelled on the server as well.

  • Rename crandas.query to crandas.command to be consistent withserver-side implementation and to differentiate from the newcrandas.queries module

  • Add module crandas.queries providing client-side implementation ofthe task-oriented VDL query API, and use this for all queriesperformed via vdl_query. To perform queries, a block-then-pollstrategy is used where first, a blocking query with a timeout of 5seconds is performed, and if the result is not ready then, statusupdate polls are done at a 1 second interval

• All column types now support missing values

  All ctypes now support a nullable flag, indicating that valuesmay be missing. It may also be specified using a question mark,e.g.varchar?.

• Progress reporting for long-running queries

  Queries that take at least 5 seconds now result in a progress barbeing displayed that estimates the progress of the computation.

  To enable this for Jupyter notebooks, note that crandas should beinstalled with the notebook dependency flag, see below.

• Various memory improvements for both server and client

• Large data uploads and downloads are now automatically chunked

  Uploads are processed in batches of sizecrandas.ctypes.ENCODING_CHUNK_SIZE.

• Added a date column type

  Dates can now be encoded using the date ctype.

  • Dates limited between 1901/01/01 - 2099/12/31 for leap yearreasons

  • Ability to subtract two dates to get number of days and add daysto a date

  • All comparison operators apply for date

  • Created functions for year, month, day and weekday

  • Able to group over dates, merge and filter

  • New ctype DateCtype converts strings (throughpd.to_datetime) and python dates (datetime.date,datetime64 and pd.timestamp) into crandas dates

  • Helper subclass of CSeries _DT allows for pandas-stylecalling of date retrieval functions (col.dt.year) andstandard calls (col.year).

Crandas¶

• New dependencies: tqdm and pyformlang

• New dependency flag: notebook, for features related to Jupyternotebooks. Use pip install crandas[notebook] to install these.

• Dependency urllib3 is updated to ensure ‘assert_hostname = False’does work as expected

• Documentation updates

• Recording or loading a new script when there is already anotherscript active now no longer gives an error, but a warning message isprinted instead.

• feat(crandas): support with_threshold for aggregation

  This adds support fore.g.table["column"].with_threshold(10).sum(). Before thischange, with_threshold() was only supported for filteringoperations, e.g. table[filter.with_threshold(5)], and not foraggregation operations (min, max, sum, etc.).

  Note that the alternative that worked beforetable["column"].sum(threshold=5) is still supported, for bothaggregation and filtering operations.

  Minor change: supplying both with_threshold() and a thresholdargument now raises a ValueError instead of a TypeError when theseare different.

• implement setter for base_path

  The crandas Session objects now supports setting base_path toeither a string, a Path, or None. Retrieving the property will alwaysreturn a Path.

• Fix problem where calling size() on a groupby object would fail forint32 columns

• Improved message for auto-determined bounds

  • Collect all auto_bounds warnings from a data upload into a singlewarning message

  • Allow to set auto_bounds globally in crandas.base.session