Saturday, February 25, 2023

3.0.0 - Internal Overhaul

It has been over year since the last major release, and in the meantime a whole host of improvements have landed with some some breaking changes along the way.  The general theme of this work has been to reduce edge cases, and to align with external standards where possible.  The average application will see very little impact to existing code bases, as the changes tend to impact the core of the framework more than anything else.

What has changed?

Breaking Changes

  • Compilation has been reworked from the ground up. This has been one of the biggest changes in this release, and has resulted in some key benefits, primarily support for NPM and Yarn Workspaces, proper support for Ecmascript Modules, and integration with Rollup for fully tree-shakeable builds.  An added benefit is that the Travetto Github repo is now the first consumer of the monorepo support.  This also has the addede benefit of providing better integration with the VS Code plugin.  The new compiler now helps to orchestrate access across multiple invocations to ensure only one build is running at a time.
  • Logging has moved to Dependency Injection. The standard logging support will continue to work without change, but custom loggers are now provided via the Dependency Injection system.  This allows for complex logging solutions that can leverage existing services at runtime.  This also means services can now inject specific loggers as needed and leverage multiple loggers as needed.
  • Config has moved to Dependency Injection.  While the general functionality of configurations is mainly unchanged, the Configuration module now supports providing configuration content via the Dependency Injection system.  This means remote/external/complex configurations can now be used for configuring services and runtime behavior. 
    A true breaking change for Config is that it no longer attempts to map all environment variables into the configuration data.  This was rife with accuracy and security concerns and so it has been dropped in favor of a new field decorator.  The new decorator, @EnvVar("PROP"), allows for mapping specific environment variables with explicit understanding and execution.  
  • Extensions within Modules have been externalized.  This was a hard requirement for the new compiler, as the compiler assumes idemptotent compilation, and the extension system relied on file-level directives that could change depending on the dependency tree for a given module.  In lieu of the in-module extensions, the affected code has now been externalized to explicit modules:
    • Auth-model - Holds model for auth persistence with the model framework
    • Auth-rest-jwt - Support for auth-rest and jwt tokens
    • Auth-rest-session - Support for auth-rest and session integration
    • Auth-rest-context - Support for exposing the auth user into the request context support
    • Auth-passport - Clear support for passport, and handles pulling the correct dependencies
    • Email-nodemailer - Clear support for nodemailer, and handles pulling the correct dependencies
    • Rest-aws-lambda - Extracted all aws lambda code and dependencies, and can be pulled in as needed
    • Rest-express-lambda - rest-express + rest-aws-lambda + necessary deps
    • Rest-koa-lambda - rest-koa + rest-aws-lambda + necessary deps
    • Rest-fastify-lambda - rest-fastify + rest-aws-lambda + necessary deps
    • Schema-faker - Clear support for faker, and handles pulling the correct dependencies

    This also has the added benefit of removing most of the external peer dependencies, so by installing one of the modules listed above, you will get the appropriate dependencies without explicit knowledge of the underlying implementation.  
  • Resources Overhaul. With the new compilation, and support for rollup, a hard requirement is that the runtime of programs needs to assume as little as possible about direct filesystem access. In honoring that, the Resource patterns have been simplified, and no longer acts a global registry of various file searches.  It is now an opt-in pattern, and is constrained as much as possible. For the consuming services, this has resulted in no visible changes in features, but has provided stronger guarantees around runtime behavior.
  • VS Code Extension.  The VS Code extension for Travetto has been reworked from the ground up to work with all the changes listed above, and specifically the compiler changes.  The latest version of the extension is no longer compatible with the 2.x.x framework line.  Functionally the plugin should appear nearly identical, and behave the same, with a perceived performance boost.
  • Pack + Rollup.  The pack module has been completely gutted, and is only similar in name at this point.  The general usage patterns should be equivalent, but the underlying ouptut, operations, and configuration options are completely changed.  The added benefit here is that with Rollup being a known/standard tool, the output quality should improve with far less maintenance overhead.
  • JSX Support. As a general step forward, the framework now supports transpiling .tsx files.  The compiler makes no assumptions about the JSX runtime that will be used, but is now able to tie into Typescripts general support for JSX.
    • The Doc module (gnerally used internally) and all framework documentation have been moved over to JSX, using a simplistic runtime to support the documentation module.  

Major Changes

  • Rest Interceptors and Configuration. With this release, it is now possible to configuration interceptor behavior on a global and per route pattern, including fully enabling and disabling. This pattern is now standardized, and is supported by all built in interceptors.
  • Typescript 4.9+. The framework has moved to Typescript 4.9 with a strong eye towards the 5.0 release.  The new ES Decorator proposal is incoming, but is still lacking parameter support which will be a blocking feature from switching.  Typescript 5.3 will most likely provide the necessary functionality for switching away from the legacy decorator functionality.
  • ESLint as a module. There is now a module for integrating with ESLint and it's new flat configuration mechanism.  Additionally this module provides the ability to write your own lint rules in within your codebase.
  • Mono Repo support. As noted above, yarn/npm workspaces are now fully supported, and the framework is the primary dogfooder of that logic.  Additionally there is a new module '@travetto/mono-repo' that provides support for versioning and publishing mono repo projects.  
  • Ecmascript Module support. ESM support is now fully baked into the framework, including proper support in the pack module.  Currently the only caveat within the framework is the hot-reloading functionality rely on import hooks that are only well supported in CommonJS.  There is active work within Node to shore up the API for ESM module hooking, and when that lands ESM should have complete parity with CommonJS.
  • Enhanced Terminal Support. The framework has embraced the ability to interact with terminals, including progress information (streaming output, progress status, etc). Additionally the module has provided better color integration with the terminal including different color depths, and detection of light vs dark backgrounds.
  • Compiler/Build/VSCode Logging. One of the stronger quality of life improvements within this release is improved logging for the various tooling touch points.  The compiler and the vscode extension now provides structured and more detailed logs.  This also provides a solid foundation for building higher quality bug reports.
  • Test Streaming. Tests now support a new output format 'tap-streamed' which will show a progress bar of the entire test run, and will show error/output information above the progress bar as needed.  This is another nice quality of life improvement to allow a more summary-oriented test format.
  • Test Execution. Previously test execution relied on unloading code at runtime to minimize latency on running tests quickly.  This code was extremely brittle, and incompatible with ESM.  This code has been completely dropped and now the test runner will spawn a new instance on each execution.  There may be room for improvement, but the general testing performance is unchanged, and is actually more stable due to the decreased complexity in execution.

Whats Next?

The code will run through a few minor releases for stability, but hopefully this release can sit for a while as this is has been a pretty substantial investment.  That being said, the short list of what's next are still in line with the previous release's future looking goals:

Heap Snapshotting

With the ESM support on the horizon (and the impact it has on unloading modules at runtime), there is real impact on the testing framework performance. With the ongoing work for heap snapshot support in V8/NodeJS, this opens the door for minimizing the performance issues, while dramatically simplifying the testing/loading code.  

ES Decorators

As stated above, the expectation is that the full ES Decorator proposal will land by end of this year.   This should be a pretty minor change within the framework, but will produce much cleaner output with much less dependency on tslib. With the full ES Decorator propsal implemented, the framework will also be able to leverage functions as first class citizens.  This should allow for less code in situtations where the wrapping class was unnecessary.