Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #254 - special benchmark for TypeBox #255

Merged
merged 8 commits into from
Oct 21, 2022
Merged

Conversation

samchon
Copy link
Owner

@samchon samchon commented Oct 21, 2022

1st completion for TypeBox benchmark.

@sinclairzx81 Please check this PR and tell me if something is wrong.

@samchon samchon added the enhancement New feature or request label Oct 21, 2022
@samchon samchon self-assigned this Oct 21, 2022
@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

Oops, you revived ajv in json-schema type without validation. A commit would be sent more.

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

Anyway, ignoring the ajv, that's result. Your library is much faster in is() function.

It would be better to adding below image on your repo.

Anyway, I'll add validation rule of other libraries at tomorrow.

image

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

About other functions, but they're slow.

image

image

image

@sinclairzx81
Copy link
Contributor

Heya.

See below for schema fixes. These should be compatible with both Ajv and TypeBox. Note that additionalProperties: false is essentially the same as the union specialization algorithm. It's just that by default, JSON Schema is permissive of extra properties, so you do need to explicitly opt out of them. See http://json-schema.org/understanding-json-schema/reference/object.html#additional-properties for more info.

Feel free to update the other schemas with additionalProperties: false to bring validation checks functionality inline with TSON. Also, see here for the general performance degradation when using additionalProperties: false (strictAssert). Performance does drop quite a bit for strict property assertion.

Fixes below.

TypeBoxObjectUnionImplicit.ts
import { Type } from "@sinclair/typebox";
import { TypeCompiler } from "@sinclair/typebox/compiler";

const Point = Type.Object(
 {
     x: Type.Number(),
     y: Type.Number(),
 },
 { additionalProperties: false },
);
const Circle = Type.Object(
 {
     centroid: Point,
     radius: Type.Number(),
 },
 { additionalProperties: false },
);
const Line = Type.Object(
 {
     p1: Point,
     p2: Point,
 },
 { additionalProperties: false },
);
const Triangle = Type.Object(
 {
     p1: Point,
     p2: Point,
     p3: Point,
 },
 { additionalProperties: false },
);
const Rectangle = Type.Object(
 {
     p1: Point,
     p2: Point,
     p3: Point,
     p4: Point,
 },
 { additionalProperties: false },
);
const Polyline = Type.Object(
 {
     points: Type.Array(Point),
 },
 { additionalProperties: false },
);
const Polygon = Type.Object(
 {
     outer: Polyline,
     inner: Type.Array(Polyline),
 },
 { additionalProperties: false },
);

const Union = Type.Union([
 Point,
 Line,
 Triangle,
 Rectangle,
 Polyline,
 Polygon,
 Circle,
]);

export const __TypeBoxObjectUnionImplicit = Type.Array(Union);
export const TypeBoxObjectUnionImplicit = TypeCompiler.Compile(
 __TypeBoxObjectUnionImplicit,
);
TypeBoxUltimateUnion.ts
import { TSchema, Type } from "@sinclair/typebox";
import { TypeCompiler } from "@sinclair/typebox/compiler";

const Attribute = {
 description: Type.Optional(Type.String()),
 "x-tson-metaTags": Type.Optional(Type.Array(Type.Any())),
 "x-tson-jsDocTags": Type.Optional(Type.Array(Type.Any())),
};

const Unknown = Type.Object({}, { additionalProperties: false });
const NullOnly = Type.Object(
 {
     type: Type.Literal("null"),
     ...Attribute,
 },
 { additionalProperties: false },
);

const Atomic = (literal: string, type: () => any) => {
 return Type.Object(
     {
         type: Type.Literal(literal),
         nullable: Type.Boolean(),
         default: Type.Optional(type()),
         ...Attribute,
     },
     { additionalProperties: false },
 );
};

const Constant = (literal: string, type: () => any) =>
 Type.Intersect(
     [
         Atomic(literal, type),
         Type.Object({
             enum: Type.Array(type()),
         }),
     ],
     { additionalProperties: false },
 );

const Array = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         type: Type.Literal("array"),
         items: schema,
         nullable: Type.Boolean(),
         ...Attribute,
     },
     { additionalProperties: false },
 );

const Tuple = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         type: Type.Literal("array"),
         items: Type.Array(schema),
         nullable: Type.Boolean(),
         ...Attribute,
     },
     { additionalProperties: false },
 );

const Reference = Type.Object(
 {
     $ref: Type.String(),
     ...Attribute,
 },
 { additionalProperties: false },
);

const RecursiveReference = Type.Object(
 {
     $recursiveRef: Type.String(),
     ...Attribute,
 },
 { additionalProperties: false },
);

const OneOf = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         oneOf: Type.Array(schema),
         ...Attribute,
     },
     { additionalProperties: false },
 );

const ObjectDef = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         $id: Type.Optional(Type.String()),
         type: Type.Literal("object"),
         nullable: Type.Boolean(),
         properties: Type.Record(Type.String(), schema),
         patternProperties: Type.Optional(
             Type.Record(Type.String(), schema),
         ),
         additionalProperties: Type.Optional(schema),
         required: Type.Optional(Type.Array(Type.String())),
         description: Type.Optional(Type.String()),
         "x-tson_jsDocTags": Type.Optional(Type.Array(Type.Any())),
         $recursiveAnchor: Type.Optional(Type.Boolean()),
     },
     { additionalProperties: false },
 );

const Components = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         schemas: Type.Record(Type.String(), ObjectDef(schema)),
     },
     { additionalProperties: false },
 );

const Application = <T extends TSchema>(schema: T) =>
 Type.Object(
     {
         schemas: Type.Array(schema),
         components: Components(schema),
         purpose: Type.Union([Type.Literal("swagger"), Type.Literal("ajv")]),
         prefix: Type.String(),
     },
     { additionalProperties: false },
 );

const Schema = Type.Recursive((schema) =>
 Type.Union([
     Atomic("boolean", () => Type.Boolean()),
     Atomic("integer", () => Type.Number()),
     Atomic("bigint", () => Type.Number()),
     Atomic("number", () => Type.Number()),
     Atomic("string", () => Type.String()),
     Constant("boolean", () => Type.Boolean()),
     Constant("integer", () => Type.Number()),
     Constant("bigint", () => Type.Number()),
     Constant("number", () => Type.Number()),
     Constant("string", () => Type.String()),
     Array(schema),
     Tuple(schema),
     Reference,
     RecursiveReference,
     OneOf(schema),
     Unknown,
     NullOnly,
 ]),
);

export const __TypeBoxUltimateUnion = Type.Array(Application(Schema));
export const TypeBoxUltimateUnion = TypeCompiler.Compile(
 __TypeBoxUltimateUnion,
);

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

You mean that when no dynamic property exists, must assign additionalProperties: false?

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

I'll remake benchmark program on Saturday, but it would be measured by another computer (surface pro8)

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

@sinclairzx81 Wait, if you utilize additionalProperty: false, wouldn't TypeBox be much slower by dynamic iteration (Object.keys()) ?

@sinclairzx81
Copy link
Contributor

sinclairzx81 commented Oct 21, 2022

Wait, if you utilize additionalProperty: false, wouldn't TypeBox be much slower by dynamic iteration (Object.keys()) ?

Yes! there is a performance cost to additionalProperties: false. See here for TB's attempts to optimize around it. Pretty much all the validators pay a performance cost for key enumeration, and unfortunately i think it's unavoidable (i wish v8 could provide an optimized code path for it) :(

But just on this (and TSON's default strict excess property checks), have you considered the following as it relates to structural checks in TypeScript?

interface Vector3 {
  x: number,
  y: number,
  z: number
}
interface Vector2 {
  x: number,
  y: number,
}

function add(a: Vector2, b: Vector2): Vector2 {
  return { x: a.x + b.x, y: a.y + b.y }
}

// ------------------------------------------------------
// Allowed: Structural / Polymorphic Assignment
// ------------------------------------------------------

const a: Vector3 = { x: 0, y: 0, z: 0 }
const b: Vector3 = { x: 0, y: 0, z: 0 }
add(a, b) // ok !

// ------------------------------------------------------
// Not Allowed: (additionalProperties: false)
// ------------------------------------------------------

add( 
  { x: 0, y: 0, z: 0 }, // <-- error !
  { x: 0, y: 0, z: 0 }
)

Was wondering if TSON could detect type usage and only apply the union specialization algorithm in cases where direct assignment wasn't possible (inline the TS structural assignment - it has a term, but i forget the name of it). Ill try look it up.

Update: I think TS lumps these assignment rules under "Excess Property Checks" Link Here, but I think the general term is "Construction vs Assignment" where Construction + Assignment leads to error. But in the general case, allowing for excess properties can be quite useful in the structural sense (and they're faster to check!!).

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

Wow, I found a miracle performance on your TypeBox.

How ObjectUnionImplicit can be faster than ObjectUnionExplicit even when optional property exists?

Can you explain me the reason why?

Components typescript-json typebox ajv io-ts class-validator zod
object (hierarchical) 108210.18756895918 184181.2811519291 74299.2341356674 9138.838143953233 62.60177311380495 401.32304299889745
object (recursive) 66304.31602048282 84578.3940547399 Failed 4415.937272064187 40.39463886820551 71.25581395348837
object (union, explicit) 15859.748072946042 11516.663631396832 1117.5094475436385 3193.8794209272496 16.305354826755604 32.13003213003213
object (union, implicit) 11555.311355311356 76721.90013091452 4082.611100866679 Failed 16.37852593266606 45.85798816568047
array (recursive) 6734.898727370497 6862.402380066939 Failed 466.87580734452854 3.3815517565282733 9.095971783924263
array (union, explicit) 3723.4714737037725 1976.873265494912 Failed 391.80962921970115 7.663551401869159 2.822732404968009
array (union, implicit) 3689.2798839107563 2275.968710205567 Failed 460.33519553072625 8.64397622906537 3.549411544928078
ultimate union 548.9229645856152 252.66582324236398 Failed Failed Failed 0.3459609064175748

@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

Well, I changed data generation function of ObjectUnionImplicit a little bit and failed to validating.

I didn't understand how ObjectUnionExplicit can be slower than ObjectUnionImplicit. The answer was it's just a bug occured from TypeBox. I think it would better to find the reason why and fix the bug. For your convenient debugging, I made a javascript file archived generated codes from your TypeBox library.

Anyway, your TypeBox is really fast when validing is() function targetting to sole objects. It's really awesome and I need to learn from your code. Below chart is a present for you. How about attach below graph on your repo with the special title The fastest validator in the word.

image

The fastest validator in the word, measured by typescript-json on Intel i5-1135g7, Surface Pro 8

Also, just understand me that I should utilize below graph for advertising.

image

@samchon samchon merged commit 8326ef8 into master Oct 21, 2022
@samchon
Copy link
Owner Author

samchon commented Oct 21, 2022

@sinclairzx81, this is not related to this benchmark PR.

Anyway, when I present this project on a seminar, someone suggested me that shorter name would be much better and recommended me to rename typescript-json to typia. As a forerunner than me, how do you think about that idea? To be a famous library, it would better to rename to be shorter?

@samchon samchon deleted the features/benchmark branch October 21, 2022 16:48
@sinclairzx81
Copy link
Contributor

sinclairzx81 commented Oct 22, 2022

@samchon Heya. You might want to fix up that ObjectUnionImplicit test. If you need properties to be optional (as in missing), then you will need to use Type.Optional() instead. Type.Undefined() will only test the property value is undefined, it doesn't test for the properties existence (noting there has been data changes here since yesterday)

Current Implementation

// { prop: undefined | null | number } - expects `prop` to exist
prop: Type.Union([Type.Undefined(), Type.Null(), Type.Number()])

Fixed Implementation

// { prop?: null | number } - prop is optional
prop: Type.Optional(Type.Union([Type.Null(), Type.Number()]))

The following fixes up the test

TypeBoxObjectImplicitUnion.ts
import { Type } from "@sinclair/typebox";
import { TypeCompiler } from "@sinclair/typebox/compiler";

const Point = Type.Object(
  {
      x: Type.Number(),
      y: Type.Number(),
      slope: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Line = Type.Object(
  {
      p1: Point,
      p2: Point,
      distance: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Triangle = Type.Object(
  {
      p1: Point,
      p2: Point,
      p3: Point,
      width: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
      height: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
      area: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Rectangle = Type.Object(
  {
      p1: Point,
      p2: Point,
      p3: Point,
      p4: Point,
      width: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
      height: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
      area: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Polyline = Type.Object(
  {
      points: Type.Array(Point),
      length: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Polygon = Type.Object(
  {
      outer: Polyline,
      inner: Type.Optional(Type.Array(Polyline)),
      area: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);
const Circle = Type.Object(
  {
      centroid: Type.Optional(Point),
      radius: Type.Number(),
      area: Type.Optional(Type.Union([Type.Null(), Type.Number()])),
  },
  { additionalProperties: false },
);

const Union = Type.Union([
  Point,
  Line,
  Triangle,
  Rectangle,
  Polyline,
  Polygon,
  Circle,
]);

export const __TypeBoxObjectUnionImplicit = Type.Array(Union);
export const TypeBoxObjectUnionImplicit = TypeCompiler.Compile(
  __TypeBoxObjectUnionImplicit,
);

For the purposes of Ajv testing, just note JSON schema doesn't have a specification for undefined values (as these are not serializable to JSON). TypeBox does provide compiler support for undefined, but its listed as non standard. See https://github.com/sinclairzx81/typebox#types-extended for non-standard types that can't be used with Ajv.

Will submit a PR to fix this up.

@sinclairzx81
Copy link
Contributor

someone suggested me that shorter name would be much better and recommended me to rename typescript-json to typia. As a forerunner than me, how do you think about that idea?

Typia sounds like a good name, you should reserve it!

To be a famous library, it would better to rename to be shorter?

I don't really select libraries based on what they're named (so I'm probably the wrong person to ask). I guess a shorter name can help the success of a library, but I'm mostly looking for functionality, documentation and ease of use first and foremost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants