The Lab of Portfolio Pipelines: Comparing Investment Platform Workflow Logic

This overview reflects widely shared professional practices as of May 2026; verify critical details against current official guidance where applicable.

The Stakes: Why Workflow Logic Matters in Portfolio Pipelines

Every investment platform, whether a robo-advisor or a full brokerage, relies on a pipeline—a sequence of steps that transforms raw market data into a constructed portfolio. The logic that governs this pipeline determines how quickly your orders execute, how well your risk tolerance is reflected, and ultimately, how your returns compare to your benchmarks. Yet most investors never see the engine; they only experience the output. This lack of transparency can lead to mismatched expectations when, for example, a platform claims "real-time rebalancing" but actually runs nightly batch jobs. In this lab, we crack open the black box.

The Hidden Cost of Pipeline Design

Consider a typical scenario: a user sets a target allocation of 60% equities and 40% bonds. An unexpected market dip causes equities to drop to 55%. A platform with a batch pipeline might only detect this drift at the end of the day, triggering a rebalance the next morning—by which time the market may have rebounded, causing unnecessary trades and tax implications. In contrast, an event-driven pipeline would detect the drift within minutes and rebalance immediately, potentially capturing the rebound. The difference is not just speed; it's the logic of when and how the pipeline evaluates portfolio state. This has real consequences for volatility dampening and tax-loss harvesting efficiency.

Three Pipeline Archetypes

Through our analysis of dozens of platforms, we identify three dominant workflow logics: the Batch Processor, the Event-Driven Responder, and the Hybrid Scheduler. The Batch Processor runs all steps—data ingestion, risk scoring, optimization, trade generation—on a fixed schedule (e.g., every 15 minutes or end-of-day). It is simple to build and audit but can miss fast-moving opportunities. The Event-Driven Responder triggers each step based on data changes—a new price feed, a user deposit, a risk model update. It is more responsive but harder to debug and can become resource-intensive. The Hybrid Scheduler combines both: batch for routine tasks (e.g., daily rebalancing) and event triggers for critical events (e.g., margin calls, large deposits). Most modern platforms lean hybrid, but the exact logic varies widely.

Why This Comparison Matters for Your Portfolio

The workflow logic directly affects three key metrics: latency (time from market event to portfolio action), cost (transaction fees and tax implications from unnecessary trades), and customization (how precisely the pipeline can reflect your personal constraints). A platform optimized for low latency may sacrifice tax efficiency if it triggers too many small trades. Conversely, a cost-focused batch pipeline may let drift accumulate, increasing risk. By understanding these trade-offs, you can choose a platform that aligns with your investment horizon and tax situation. This guide will equip you with the framework to evaluate any platform's pipeline logic, using concrete examples and step-by-step comparisons.

Core Frameworks: How Pipeline Logic Works Under the Hood

To compare workflow logics, we must first understand the common components of any portfolio pipeline. Every pipeline consists of three core phases: data ingestion, analysis and decision, and execution. Data ingestion pulls in market prices, account balances, and external signals (e.g., news sentiment). The analysis phase applies risk models, rebalancing rules, and optimization algorithms. Execution generates buy/sell orders and submits them to exchanges. The workflow logic defines the control flow between these phases—whether they run sequentially, in parallel, or with feedback loops. We will examine each phase and how different logic choices affect the whole.

Data Ingestion: Polling vs. Streaming

The first fork in pipeline design is how data enters the system. Polling-based ingestion checks for new data at fixed intervals (e.g., every 5 minutes). This is simple and cost-effective but introduces latency equal to the polling interval. Streaming ingestion, using technologies like Apache Kafka or WebSockets, pushes data as it changes. This reduces latency to milliseconds but requires more infrastructure and can overwhelm downstream components if not properly throttled. In a batch pipeline, polling is natural; in an event-driven pipeline, streaming is essential. The choice here cascades: a streaming ingestion often forces the rest of the pipeline to be event-driven to avoid bottlenecks. Platforms targeting high-frequency trading or real-time risk monitoring invariably use streaming, while long-term wealth managers often stick with polling to reduce complexity and cost.

Analysis and Decision: Synchronous vs. Asynchronous

Once data is ingested, the analysis phase must decide whether to recompute portfolio metrics from scratch or incrementally. Synchronous analysis waits for all data to arrive before running the full optimization—this is typical in batch pipelines. Asynchronous analysis updates portions of the portfolio as new data streams in, using techniques like incremental risk decomposition. For example, a platform using asynchronous logic might update the equity risk contribution whenever a new price tick arrives, without recomputing the entire covariance matrix. This reduces computational load but can lead to inconsistencies if different parts of the portfolio are updated at different times. The choice between synchronous and asynchronous is a trade-off between correctness (consistency) and speed (freshness). Most hybrid pipelines use synchronous for daily rebalancing and asynchronous for intraday alerts.

Execution: Immediate vs. Scheduled

The final phase, execution, also has a logic fork. Immediate execution sends orders to the market as soon as the analysis phase produces them. This minimizes latency but can lead to market impact if orders are large. Scheduled execution collects orders over a window and submits them at a preset time (e.g., at the close of the next 15-minute bar). This reduces market impact and allows for netting of buy and sell orders, lowering transaction costs. However, it introduces a delay that may be unacceptable for time-sensitive strategies. A platform's workflow logic must decide how to balance these factors. For example, a tax-loss harvesting pipeline might schedule execution to avoid same-day wash sales, while a momentum strategy might require immediate execution to capture price trends. The decision is rarely binary; many platforms use a tiered approach: immediate for small orders, scheduled for large ones.

Comparing Three Platforms: A Conceptual Lab Setup

To make these frameworks concrete, we will compare three hypothetical but representative platforms: AlphaBatch (pure batch), OmegaStream (pure event-driven), and SigmaHybrid (hybrid). AlphaBatch uses polling every 15 minutes, synchronous analysis, and scheduled execution at the top of each hour. OmegaStream uses streaming data, asynchronous incremental analysis, and immediate execution. SigmaHybrid polls every 5 minutes for routine data but streams critical signals (e.g., volatility spikes); it uses synchronous analysis for daily rebalance and asynchronous for intraday alerts, and schedules execution for routine trades but allows immediate execution for emergency adjustments. This setup will serve as our lab for the rest of the guide, allowing us to trace how each logic choice affects real-world performance.

Execution: Workflows and Repeatable Processes

Moving from theory to practice, this section details the exact workflows that AlphaBatch, OmegaStream, and SigmaHybrid follow for a typical rebalance event. We will walk through each step, highlighting where the logic diverges and what that means for the end user. By the end, you should be able to map any platform's workflow to one of these archetypes and predict its behavior under different market conditions.

AlphaBatch: The Scheduled Rebalance

AlphaBatch's workflow begins with a timer set to every 15 minutes. At the 15-minute mark, the ingestion module polls all data sources (prices, account balances, risk model outputs). If no new data is available (e.g., market closed), the pipeline skips this cycle. Once data is collected, the analysis module runs a full portfolio optimization using the latest data. This includes recalculating the efficient frontier, checking drift from target allocations, and generating a list of trades. The execution module then queues these trades and submits them at the next scheduled execution window (the top of the hour). For a user, this means that if a market event occurs at 10:07 AM, it will not be reflected in their portfolio until at least 11:00 AM—and potentially later if the rebalance is small enough to be deferred. The advantage is predictability: the user knows exactly when rebalances happen, and the platform can batch trades to minimize fees. The disadvantage is latency: during volatile periods, the portfolio can drift significantly between scheduled checks.

OmegaStream: The Continuous Adjustment

OmegaStream's workflow is entirely event-driven. The moment a new price tick arrives for any held asset, the ingestion module pushes it to the analysis module. The analysis module uses an incremental risk model: it updates the asset's contribution to portfolio risk without recomputing the entire covariance matrix. If the drift exceeds a configurable threshold (e.g., 1% of target allocation), it immediately generates a trade order. The execution module sends this order to the market with no delay. For a user, this means their portfolio is always within 1% of target, even during rapid market moves. However, this can lead to a high number of small trades, increasing transaction costs and potentially triggering wash sales if not carefully managed. OmegaStream addresses this by using a "deadband"—a range around the target where no action is taken—and by netting trades over a short window (e.g., 1 minute) before submitting. But the core logic remains: react immediately to any significant change.

SigmaHybrid: The Best of Both Worlds

SigmaHybrid attempts to combine the strengths of both approaches. Its workflow uses a primary batch loop that runs every 5 minutes for routine data ingestion and analysis. During this loop, it performs a full portfolio optimization (synchronous analysis) and schedules trades for execution at the next 30-minute window. However, it also has a secondary event-driven loop for critical signals. For example, if a real-time volatility index exceeds a threshold, the ingestion module streams that signal directly to an incremental risk model. If the incremental model detects that portfolio risk has crossed a predefined limit, it triggers an immediate rebalance, bypassing the scheduled execution window. This hybrid approach allows SigmaHybrid to maintain low transaction costs during normal times while providing rapid response during crises. The trade-off is complexity: the platform must manage two parallel pipelines and resolve conflicts (e.g., what if the batch loop and event loop both generate trades for the same asset?). SigmaHybrid resolves this by giving the event loop priority and canceling any conflicting scheduled trades.

Step-by-Step: How to Evaluate a Platform's Workflow

When you are considering a platform, you can reverse-engineer its workflow logic by asking a few key questions. First, check the rebalancing frequency: does the platform rebalance on a fixed schedule (daily, weekly) or continuously? Second, look at the trade execution: are orders submitted immediately or batched? Third, examine how the platform handles tax implications: does it track wash sales across trades? Fourth, test the platform's response to a simulated market shock: how long does it take for a drift to be corrected? By combining these observations, you can classify the platform into one of the three archetypes. For example, many popular robo-advisors use a batch pipeline with daily rebalancing, while newer platforms targeting active traders often use event-driven logic. The key is to match the pipeline's strengths to your own investment style: if you are a buy-and-hold investor, batch logic is probably sufficient; if you trade frequently or use leverage, event-driven may be worth the higher cost.

Tools, Stack, Economics, and Maintenance Realities

Implementing a portfolio pipeline is not just about logic design; it also involves choosing the right tools, managing costs, and maintaining the system over time. This section explores the technology stacks behind the three archetypes, the economic trade-offs, and the operational realities that platform teams face. Understanding these factors will help you appreciate why platforms make the design choices they do—and what those choices mean for you as a user.

Technology Stack: From Databases to Message Queues

Batch pipelines like AlphaBatch typically rely on relational databases (e.g., PostgreSQL) for storage and cron jobs for scheduling. The analysis module might be a Python script that runs at fixed intervals, querying the database for the latest data and writing results back. This stack is cheap to build and maintain, but it does not scale well under high data velocity. Event-driven pipelines like OmegaStream use streaming platforms (e.g., Apache Kafka or Amazon Kinesis) to ingest data, in-memory data grids (e.g., Redis) for state management, and microservices triggered by events. This stack is more expensive and complex but can handle millions of events per second. Hybrid pipelines like SigmaHybrid often use a combination: a traditional database for the batch loop and a streaming platform for the event loop, with a coordination layer (e.g., Apache Airflow) to manage dependencies. The choice of stack directly affects the platform's cost and its ability to add new features. For example, adding a new data source to a batch pipeline might require modifying a cron job and a SQL query; in an event-driven pipeline, it might require a new microservice and a new Kafka topic.

Economic Trade-offs: Latency vs. Cost

The cost of running a pipeline is a function of compute, storage, and data transfer. Batch pipelines are cheap because they can use spot instances and shut down between runs. Event-driven pipelines require always-on infrastructure, which increases cloud costs. However, event-driven pipelines can reduce transaction costs by enabling more precise rebalancing—fewer unnecessary trades. The breakeven point depends on the portfolio size and trading frequency. For a small portfolio with infrequent trades, the extra infrastructure cost of an event-driven pipeline may outweigh any savings from reduced drift. For a large portfolio with many trades, the savings can be significant. Additionally, event-driven pipelines often require more expensive engineering talent to build and maintain. Platform teams must weigh these factors when choosing a workflow logic. As a user, you should ask: does the platform's pricing model reflect its pipeline costs? Some platforms charge a flat fee regardless of pipeline complexity, while others charge a percentage of assets under management, which may be higher for premium pipeline features.

Maintenance Realities: Debugging and Monitoring

Maintaining a pipeline is an ongoing challenge. Batch pipelines are easier to debug because each run is independent and logs are straightforward. If a batch job fails, you can rerun it with the same data. Event-driven pipelines are harder to debug because events are asynchronous and state is distributed. A single bug can cause cascading failures that are difficult to trace. Hybrid pipelines combine the worst of both: you have batch jobs that can fail and event streams that can get out of sync. Teams often invest in sophisticated monitoring and alerting systems, such as distributed tracing (e.g., Jaeger) and log aggregation (e.g., ELK stack). They also implement circuit breakers to prevent runaway processes. As a user, you may not see these maintenance struggles, but they affect platform reliability. A platform with a poorly maintained pipeline may experience outages or delayed rebalances during high-volatility periods. When evaluating a platform, check its uptime history and read user reviews about execution delays.

Real-World Example: A Maintenance Incident

Consider a scenario where a hybrid platform's event loop detects a false volatility spike due to a corrupted data feed. The event loop triggers a rebalance, generating trades that conflict with the next scheduled batch run. The coordination layer fails to cancel the batch trades, resulting in duplicate orders. The platform's risk management system catches the duplication and halts all trading, but not before some orders are executed. The user sees unexpected trades in their account and may incur tax consequences. This incident illustrates the importance of robust conflict resolution and data validation in hybrid pipelines. The platform team must implement data quality checks at the ingestion layer and have a clear priority scheme for overlapping trade signals. For the user, this means choosing a platform that transparently communicates how it handles such edge cases.

Growth Mechanics: Traffic, Positioning, and Persistence

For platform teams, the pipeline logic is not just a technical decision—it is a strategic one that affects user acquisition, retention, and scalability. This section explores how different workflow logics support growth, how platforms position their pipeline features in the market, and what persistence strategies they use to keep users engaged. Understanding these mechanics can help you, as an investor, see beyond marketing claims and evaluate the real value of a platform's pipeline.

User Acquisition: The Speed Advantage

Event-driven pipelines often serve as a differentiator for platforms targeting active traders and high-net-worth individuals. Marketing materials emphasize "real-time rebalancing" and "instant risk alerts" to attract users who value responsiveness. In contrast, batch pipelines are positioned as "low-cost" and "simple," appealing to buy-and-hold investors who prioritize fees over speed. The choice of workflow logic directly shapes the platform's brand and target audience. For example, a platform that uses a hybrid pipeline might market itself as offering "the best of both worlds"—low fees with the ability to respond to market crises. This positioning can be powerful in a crowded market, but it must be backed by actual performance. If a platform claims real-time rebalancing but actually uses a batch pipeline with a 15-minute polling interval, users will eventually notice and churn. Transparency is key to long-term trust.

User Retention: Consistency and Predictability

Retention depends on the platform consistently meeting user expectations. Batch pipelines offer predictability: users know exactly when rebalances occur and can plan accordingly. This predictability can be a retention driver for users who dislike surprises. Event-driven pipelines, while responsive, can be unpredictable—a user might log in to find unexpected trades. Over time, this can erode trust if the user does not understand the logic. Hybrid pipelines aim to provide predictability during normal times and responsiveness during crises, but they require clear communication. Platforms often send push notifications or emails explaining why a trade was made, helping users understand the pipeline's actions. Persistence strategies also include educational content that explains the pipeline logic, turning a potential source of confusion into a value-add. For example, a platform might publish a blog post titled "How Our Rebalancing Engine Works" to demystify the process.

Scalability: Handling Growth

As a platform grows, its pipeline must scale to handle more users, more assets, and more frequent data. Batch pipelines scale horizontally by adding more compute nodes for the batch jobs, but they hit a ceiling when the batch window becomes too short to process all users' portfolios. Event-driven pipelines scale better because they can process events in parallel, but they require careful partitioning to avoid hotspots. Hybrid pipelines offer the most flexibility: they can use batch for the majority of users and event-driven for premium users, or they can scale the batch loop to handle increased load while using the event loop only for critical signals. However, scaling a hybrid pipeline is complex because the two loops must remain coordinated. Many platforms eventually migrate to a fully event-driven architecture as they grow, accepting the higher cost for the scalability benefits. As a user, you may notice that a platform's performance degrades during market open when many users log in; this is often a sign that the pipeline is struggling to scale.

Persistence in Feature Development

Finally, the pipeline logic affects how quickly a platform can add new features. Batch pipelines are easier to extend because you can add a new step to the batch job without worrying about event ordering. Event-driven pipelines require careful design to ensure that new events are processed in the correct sequence. Hybrid pipelines can be extended in either loop, but changes to one loop may affect the other. Platform teams must invest in testing and monitoring to ensure that new features do not break existing workflows. For users, this means that platforms with simpler pipeline logic may be able to roll out new features faster, while those with complex logic may be slower but more reliable. When evaluating a platform, consider its track record of feature releases and whether it communicates changes to its pipeline logic.

Risks, Pitfalls, and Mistakes with Mitigations

Even the best-designed pipeline can fail if common pitfalls are not addressed. This section catalogs the most frequent mistakes platform teams make when designing workflow logic, along with practical mitigations. As an investor, being aware of these pitfalls will help you spot warning signs in a platform's behavior and ask the right questions before committing your capital.

Pitfall 1: Over-Optimizing for Latency

A common mistake is to design a pipeline that minimizes latency at all costs, ignoring transaction costs and tax implications. For example, an event-driven pipeline that rebalances on every 0.5% drift may generate hundreds of trades per year, each incurring a commission and potentially triggering short-term capital gains. The result is lower net returns despite the portfolio staying perfectly aligned with the target. Mitigation: Implement a deadband (e.g., 2% drift) and a minimum trade size to avoid unnecessary trades. Also, use tax-lot accounting to select the most tax-efficient shares to sell. As a user, ask the platform about its rebalancing thresholds and whether it supports tax-loss harvesting. If the platform cannot provide clear answers, it may be over-optimizing for latency.

Pitfall 2: Ignoring Data Quality

Both batch and event-driven pipelines are vulnerable to bad data. A single erroneous price tick can trigger a cascade of incorrect trades. In batch pipelines, the damage is limited to one batch cycle, but in event-driven pipelines, the error can propagate quickly before it is detected. Mitigation: Implement data validation at the ingestion layer, such as checking that a price is within a reasonable range of its moving average. Use outlier detection algorithms to flag suspicious data points and hold them for manual review before acting on them. Also, maintain a back-up data source to cross-validate. As a user, you can check whether the platform has a policy for correcting trades made in error. Some platforms guarantee to reverse trades caused by data errors, which is a sign of robust data quality processes.

Pitfall 3: Poor Error Handling in Event-Driven Pipelines

Event-driven pipelines are particularly prone to errors because of their asynchronous nature. A failed event can cause the pipeline to stall or skip important updates. For example, if a market data feed goes down, an event-driven pipeline may not receive any events and thus take no action, even as the market moves. Mitigation: Implement a heartbeat mechanism that triggers a full refresh if no events are received within a certain time window. Also, use idempotent event processing so that duplicate events do not cause double trades. Finally, have a fallback batch pipeline that runs periodically to catch any missed events. As a user, you can test this by simulating a data outage: if the platform fails to rebalance for an extended period, its error handling may be insufficient.

Pitfall 4: Neglecting Coordination in Hybrid Pipelines

Hybrid pipelines introduce the risk of conflicting trades between the batch and event loops. Without proper coordination, a user might end up with two trades for the same asset in opposite directions, increasing costs and potentially causing a wash sale. Mitigation: Implement a trade reconciliation step that merges orders from both loops before submission. Use a global lock or a versioning system to ensure that each asset's state is consistent across loops. Also, prioritize event-driven trades and cancel any conflicting batch trades. As a user, you can ask the platform how it handles trade conflicts. If the response is vague, it may be a red flag.

Pitfall 5: Underestimating Maintenance Overhead

Platform teams often underestimate the ongoing effort required to maintain a pipeline, especially an event-driven one. Bugs in event processing logic can be hard to reproduce and fix. Over time, the pipeline can become a "spaghetti" of event handlers that are difficult to modify. Mitigation: Invest in automated testing, including integration tests that simulate real market conditions. Use event sourcing to replay past events and verify the pipeline's output. Document the workflow logic clearly so that new team members can understand it. As a user, you can gauge a platform's maintenance culture by its release notes and changelogs. Frequent updates that fix bugs and improve performance are a positive sign.

Decision Checklist and Mini-FAQ

This section provides a practical checklist to help you evaluate any investment platform's pipeline logic, followed by answers to common questions. Use this as a reference when comparing platforms or assessing your current provider.

Decision Checklist for Evaluating Pipeline Logic

• 1. Determine your latency needs: How quickly do you need your portfolio to react to market changes? If you are a long-term investor, batch logic may suffice. If you trade frequently or use leverage, consider event-driven or hybrid.
• 2. Assess your transaction cost sensitivity: If you are in a high-fee brokerage or have a small portfolio, frequent trading can erode returns. Look for platforms with wide deadbands and trade netting.
• 3. Check tax implications: If you are in a taxable account, ensure the platform supports tax-loss harvesting and can avoid wash sales. Hybrid platforms often have better tax management.
• 4. Review transparency: Does the platform explain its rebalancing logic? Can you see when trades were made and why? Transparency is a sign of confidence in the pipeline.
• 5. Test responsiveness: Create a small account and subject it to a simulated market event (e.g., a sudden drop in a held asset). Note how long it takes for the platform to rebalance.
• 6. Evaluate cost structure: Compare the platform's fees against its pipeline capabilities. A premium fee should come with premium features like real-time rebalancing.
• 7. Read user reviews: Look for mentions of trade delays, unexpected trades, or poor customer support around rebalancing issues.
• 8. Consider future needs: If you plan to increase your portfolio size or trading frequency, choose a pipeline that can scale—likely event-driven or hybrid.

Mini-FAQ: Common Questions About Pipeline Logic

Q: Is batch pipeline logic always worse than event-driven? No. Batch pipelines are simpler, cheaper, and often sufficient for buy-and-hold strategies. They also make it easier to predict and audit trades. The best logic depends on your investment style and portfolio size.

Q: Can I switch platforms easily if I don't like the pipeline logic? Yes, but be aware of tax implications. Transferring assets may trigger capital gains if you sell positions. Also, the new platform may have a different rebalancing schedule, so your portfolio may temporarily drift. Plan the switch during a low-volatility period.

Q: How can I tell if a platform is using batch or event-driven logic? Look at the platform's documentation or ask customer support: "How often does my portfolio get rebalanced?" If they give a fixed interval (e.g., daily), it's likely batch. If they say "continuously" or "within minutes of market moves," it's event-driven or hybrid. You can also test by making a deposit and seeing how quickly the platform invests the cash.

Q: Do all hybrid platforms work the same way? No. The exact coordination between batch and event loops varies. Some may prioritize event-driven trades, others may batch event-driven trades into the next scheduled window. Always ask for specifics.

Q: What if the platform's pipeline fails during a market crash? This is a critical risk. Look for platforms that have redundant infrastructure and a clear disaster recovery plan. Check if they have ever experienced an outage during high volatility and how they handled it.

Synthesis and Next Actions

We have journeyed through the lab of portfolio pipelines, dissecting the workflow logic that powers modern investment platforms. From the three archetypes—batch, event-driven, and hybrid—to the tools, economics, and pitfalls, you now have a framework to evaluate any platform's engine. The key takeaway is that there is no universally best pipeline; the right choice depends on your personal investment goals, risk tolerance, and tax situation. Batch pipelines offer simplicity and low cost, event-driven pipelines offer responsiveness, and hybrid pipelines attempt to balance both. Your job is to match the pipeline's strengths to your needs.

Immediate Next Steps

First, review your current platform's pipeline logic using the checklist above. If you are unsure, contact customer support and ask the specific questions from the FAQ. Second, consider whether your investment style has changed since you joined the platform. If you have become more active or your portfolio has grown significantly, a pipeline upgrade may be in order. Third, if you are evaluating a new platform, request a trial account and test its responsiveness during a simulated market event. Do not rely solely on marketing claims. Fourth, stay informed about pipeline innovations. As technology evolves, platforms are adopting machine learning to predict drift and optimize rebalancing schedules. These advances may tip the balance in favor of more sophisticated logic. Finally, remember that no pipeline is perfect. Even the best-designed system can fail under extreme conditions. Maintain a diversified portfolio and keep some cash reserves to weather any pipeline-related disruptions.

Final Thoughts

The lab of portfolio pipelines is a fascinating intersection of finance, software engineering, and user experience. By understanding the workflow logic, you become an informed consumer who can see beyond the user interface to the engine that drives your returns. This knowledge empowers you to ask better questions, make smarter choices, and ultimately achieve your financial goals with confidence. As the industry continues to evolve, we will keep updating this guide to reflect new practices. For now, use the tools and frameworks provided here to conduct your own lab tests and find the pipeline that works best for you.