[AWS][Kinesis] Add dimension fields for TSDB support

[constanca-m]

What does this PR do?

Add dimension fields to Kinesis datastream.

Details

To uniquely identify a Kinesis stream, we need the combination of stream name (unique per AWS region) + account ID + account region. There are no metrics split by labels, so no more dimensions should be needed. The tests with TSDB enabled and disabled did not show a change on the number of documents received.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Refer to #5864

Related issues

• Relates to [AWS Kinesis] TSDB Enablement #5864

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-05-04T07:06:25.496+0000

• Duration: 52 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	188
Skipped	4
Total	192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (15/15)	💚
Files	93.75% (15/16)	👎 -3.421
Classes	93.75% (15/16)	👎 -3.421
Methods	86.131% (236/274)	👎 -6.502
Lines	85.925% (7387/8597)	👎 -6.45
Conditionals	100.0% (0/0)	💚

[agithomas]

Review feedback shared.

[agithomas]

Please add here the reason for adding the specific field as a dimension field.Adding the reason is among the best practices for TSDB enablement.

[constanca-m]

By "here" you mean in the manifest @agithomas? It's explained in "Details" in the PR description.

[agithomas]

It can be added as the inline comment. Reference

[agithomas]

I have certain thoughts around a better way to handle aws dimensions. We have a length limitation in dimension and AWS permits 30 dimensions

If all 30 names and values are fully used to max limit, the 32KB dimension field length limitation would reach.

Can we have fingerprint processor applied on all aws dimensions and use the new field (having fingerprint) used as a dimension field?

cc @tetianakravchenko , @lalit-satapathy

[constanca-m]

Ok, I will try to add it to the document without being too confused. I don't understand the other part though, there are only 3 fields set as dimension, why would we have the need for a processor? @agithomas

[agithomas]

Please validate the proposal against

• how often aws change the dimensions of a managed service
• feasibility of including ingest pipeline and inclusion of new field only for implementing TSDB.

[constanca-m]

Sorry, I don't understand: is there a reason to create a new dimension using the dimensions.* field? The 3 fields set to dimension right now should be enough @agithomas

[agithomas]

The advantages are mentioned here.