Software Testing Basics

The AI Coding Revolution Is Here, But Are We Testing for It? As AI-assisted development reshapes how we build software, I've been thinking a lot about something that is talked about often but doesn't always get the focus it deserves: automated testing. At JPMorganChase, we're embracing AI coding tools to accelerate delivery, reduce toil, and empower our teams to focus on the work that matters, reducing cognitive load of repetitive tasks. But speed without safety is just risk in disguise. Here's what I believe every leader (and this is broader than technology) needs to consider right now: • AI writes code faster than humans can review it manually. If your testing strategy is still largely manual, you're already behind. AI-generated code can introduce subtle logic errors, security vulnerabilities, or edge-case failures that look perfectly reasonable on the surface. Automated testing is no longer a best practice, it's a non-negotiable safeguard. • Test coverage is your new quality contract. When AI is your co-developer, the test suite becomes the specification. If you can't describe expected behavior in a test, you can't trust what the AI builds. Investing in robust unit, integration, and regression testing frameworks is investing in the integrity of your entire delivery pipeline. • Shift-left testing amplifies AI's value. It doesn't slow it down. Some worry that rigorous testing will negate the speed gains from AI coding. The opposite is true. When automated tests are embedded early in the development lifecycle, AI tools can iterate faster, self-correct, and validate outputs in real time. Testing enables velocity; it doesn't constrain it. • Your teams need to evolve alongside the tools. The best teams of tomorrow won't just write code. They'll architect test strategies, evaluate AI outputs critically, and build systems that are observable and verifiable by design. We owe it to our teams to invest in this skill evolution now. At the scale we operate, serving millions of customers, the cost of a defect isn't just technical. It's trust. And trust, once broken, is hard to rebuild. AI is a force multiplier. But multiplying without a strong foundation multiplies risk just as fast as it multiplies output. Build fast. Test smarter. Ship with confidence. I'd love to hear how other leaders are thinking about quality engineering in the age of AI. What's working for your teams? #AIEngineering #SoftwareTesting

The clock code in most .NET apps is a static call your tests can never control. That one fact causes more flaky time bugs than any library ever will. DateTimeNow reads the local clock of whatever machine happens to run the code. Deploy that same service to a server in another region and your timestamps quietly shift. Nothing throws. The data just starts lying. Then daylight saving flips, and your "run at midnight" job fires twice or skips a day. Nobody wrote a bug. The clock moved and the code had no say in it. Switching to UtcNow kills the region drift - one timeline for everyone. DateTimeOffsetUtcNow goes a step further and stores the offset with the value, so you never have to guess which timezone a saved time came from. But all three are still static calls. In a test you can't say "pretend it's 11:59 PM on the night the clock rolls back." The time is whatever the machine says right now. That's the gap TimeProvider closes in .NET 8. You inject it like any other dependency, then call GetUtcNow() instead of the static one. In production it's the real system clock. In a test you hand it a FakeTimeProvider, set the exact moment you want, and even push time forward to check what happens after a delay. The rule I follow now: no DateTimeNow near anything that makes a decision. If the logic depends on what time it is, the clock gets injected. Make the clock a dependency, and time stops being something that happens to your code. It becomes something your tests get to decide.

Evaluations —or “Evals”— are the backbone for creating production-ready GenAI applications. Over the past year, we’ve built LLM-powered solutions for our customers and connected with AI leaders, uncovering a common struggle: the lack of clear, pluggable evaluation frameworks. If you’ve ever been stuck wondering how to evaluate your LLM effectively, today's post is for you. Here’s what I’ve learned about creating impactful Evals: 𝗪𝗵𝗮𝘁 𝗠𝗮𝗸𝗲𝘀 𝗮 𝗚𝗿𝗲𝗮𝘁 𝗘𝘃𝗮𝗹? - Clarity and Focus: Prioritize a few interpretable metrics that align closely with your application’s most important outcomes. - Efficiency: Opt for automated, fast-to-compute metrics to streamline iterative testing. - Representation Matters: Use datasets that reflect real-world diversity to ensure reliability and scalability. 𝗧𝗵𝗲 𝗘𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗼𝗳 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: 𝗙𝗿𝗼𝗺 𝗕𝗟𝗘𝗨 𝘁𝗼 𝗟𝗟𝗠-𝗔𝘀𝘀𝗶𝘀𝘁𝗲𝗱 𝗘𝘃𝗮𝗹𝘀 Traditional metrics like BLEU and ROUGE paved the way but often miss nuances like tone or semantics. LLM-assisted Evals (e.g., GPTScore, LLM-Eval) now leverage AI to evaluate itself, achieving up to 80% agreement with human judgments. Combining machine feedback with human evaluators provides a balanced and effective assessment framework. 𝗙𝗿𝗼𝗺 𝗧𝗵𝗲𝗼𝗿𝘆 𝘁𝗼 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲: 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗬𝗼𝘂𝗿 𝗘𝘃𝗮𝗹 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 - Create a Golden Test Set: Use tools like Langchain or RAGAS to simulate real-world conditions. - Grade Effectively: Leverage libraries like TruLens or Llama-Index for hybrid LLM+human feedback. - Iterate and Optimize: Continuously refine metrics and evaluation flows to align with customer needs. If you’re working on LLM-powered applications, building high-quality Evals is one of the most impactful investments you can make. It’s not just about metrics — it’s about ensuring your app resonates with real-world users and delivers measurable value.

One of the biggest mistakes some pentesters make is not understanding the architecture of the application they are testing. We follow methodologies, frameworks, and testing guides, often focusing more on vulnerabilities than their real business impact. As a result, we may report issues automatically without considering whether they pose real risk, expected behavior, or security layers that mitigate them. What is critical for one company may not be for another. However, testing and reporting often become routine, leading to over-reliance on tools. If Nuclei flags something as High, we might report it as such without assessing its actual impact on the company’s business model. Understanding architecture helps refine testing, ensuring medium vulnerabilities are classified correctly and low ones that pose real risk are elevated appropriately. Every company has its Achilles’ heel, making it essential to consider its business model and security layers before classifying a finding. Practical Example: Imagine finding a directory listing enabled on an internal web server. If a tool flags it as High, you might report it as critical. However, after analysis, you discover: ✅ The directory is only accessible internally. ✅ Files do not contain sensitive data. ✅ An authentication layer protects critical areas. Here, the impact is lower than suggested. Now, consider the opposite: a Low vulnerability allowing user ID enumeration in an API. If poorly designed, this could enable user enumeration in a financial app, exposing bank accounts and enabling social engineering—making it far more severe than initially classified. In a black box test, without direct access to architecture, detailed recon is crucial. Gathering information about technologies, exposed services, and authentication flows helps assess real impact. Instead of blindly trusting automated tools, pentesters should connect findings, testing exploitation chains that may escalate vulnerabilities. Understanding the application improves reporting quality and strengthens a pentester’s credibility, ensuring more strategic, business-aligned results. #cybersecurity #pentesttips #pentest #hacking #redteam

Friday .NET tip: stop hiding DateTime.UtcNow inside your code. I know. It is quick. It works. Until you need to test it. Anything with expiration dates, retries, subscriptions, cache invalidation, reminders, trial periods, token lifetimes, or "show this tomorrow" logic gets awkward fast when the current time is hardcoded. TimeProvider fixes that nicely. Production code can use TimeProvider.System. Tests can use a fake clock and move time forward without waiting, sleeping, or doing weird date math. That last bit is the important one. Instead of writing tests that depend on the actual current date and time, you control "now" directly. Want to test what happens after 30 minutes? Advance the fake clock 30 minutes. That's it. Tiny abstraction. Much nicer tests. It is one of those APIs that feels boring until the first time it removes a flaky test or lets you test a timeout properly. Pretty neat. What is the time-related test that hurt you the most? #dotnet #csharp

During the initial phase of my career in VLSI, I realised that writing Testcases is equally important as Testbench development. A Testcase in any language be it Verilog, VHDL, SystemVerilog, and UVM is not only used to verify the functional correctness and the integrity of the design but also point out areas where the Testbench could be improved. Below are the most important category of Testcases which are most critical: [1] Functional Tests --> In this type of test, the functionality or feature of an IP/module or a subsystem is verified. [2] Register-based Tests --> RW Tests, RO/WO Tests, Default Read/Hard reset Tests, Soft reset tests, Negative RO/WO Tests, Aliasing, Broadcasting, etc [3] Connectivity Tests [4] Clock and Reset Tests [5] Boot up Tests, wake up sequence, training sequence tests. For eg. In the case of DDR – MPC Training, RD DQ Calibration, Command Bus training, Write leveling, etc [6] Command and Sequence-based Tests. [7] Overlapping and Unallocated Region tests. [8] Back-to-back data transfer-based tests. [9] UPF Tests --> Power domain, Level Shifter, clock gating, voltage domain, etc [10] Code Coverage Tests --> In this test toggle, expression, branch, FSM, and conditional coverage holes are measured, and depending on the holes, tests are being written to completely exercise the DUT. [11] Functional Coverage Tests --> In these types of test categories, the functionality of DUT is being measured with the help of bins. There are several ways to do it. If there are coverage holes, more bins are coded to cover those areas, complex scenarios are covered with cross coverage, and bins of intersect functionality. [12] Assertions are basically a check against the design. Basically, these are insertion points within the design which improve the observability and debugging ability. The above are some of the categorizations of tests that need to be applied while checking a design but to achieve all the above features, testcases are broadly classified into the following two types: [1] Directed Testcase: These are the scenarios that the verification engineers can think of or can anticipate. [2] RandomTestcase: These are the scenarios where the maximum amount of bugs can be caught. The random seeds will hit many different use cases which can not be anticipated earlier and has the probability to catch the design issues. Ideally, random tests can be classified into the following two categories: [1] Corner cases --> This is the bug that is only possible to catch when many different scenarios are processed together or they overlap and the best way to catch this type of scenario is to run more repeated regression with more seeds. [2] Stress testing -->These types of tests are useful to check the performance and the scalability of the DUT under multiple concurrent activities and unpredictable scenarios. #vlsi #asic #electricalengineering #semiconductorindustry

💻 Developer: “It works perfectly on my computer.” 🧪 Tester: “That’s good. But the customer isn’t using your computer.” And that right there captures the essence of the relationship between Development and Quality Assurance. Coding proves functionality in a controlled environment. Quality Assurance validates functionality in the real world. A developer’s environment is usually optimized: • Stable internet • Clean database • Updated browser • Ideal configurations • Familiar workflows But production is different. Real users operate on: • Slow or unstable networks • Older devices and browsers • Unexpected screen resolutions • Incomplete or invalid inputs • Concurrent sessions • Unpredictable behavior QA doesn’t just test what should happen. QA evaluates what could happen. It’s not about questioning the developer’s capability. It’s about validating system resilience. A feature may work technically — but: • Does it fail gracefully? • Does it handle incorrect data securely? • Does it scale under load? • Does it maintain performance consistency? • Does it preserve user experience across devices? That is the difference between functional code and reliable software. When testers ask difficult questions, they are not creating friction — they are reducing risk. They are thinking ahead: • What breaks under pressure? • What happens outside the happy path? • What assumptions are untested? • What edge case might impact thousands of users? Users never see internal conversations. They only experience the outcome. And users don’t care where it worked. They care where it failed. Professional software teams understand that Development builds the product — but QA protects the product in reality. Behind every stable release is a tester who: • Challenged assumptions • Explored edge cases • Simulated real-world scenarios • Validated beyond the obvious That discipline is not obstruction. It is responsibility. Respect the build. Respect the validation. Respect the process that turns code into quality. #QALife #SoftwareTesting #DevVsQA #QualityAssurance #BugHunting #SoftwareDevelopment #ITLife #ProductQuality

Are you treating your test environment like an afterthought? Testing in an environment that barely resembles production sets you up for unpleasant surprises down the road. A stable, well-maintained test environment is crucial for finding issues before they hit real users. This includes having representative data, updated configurations, and monitoring tools that mirror production as closely as possible. When your environment is a patchwork of outdated databases and mismatched versions, you’re gambling on partial insights. Elevating your test environment to first-class status—complete with dedicated maintenance and robust infrastructure—empowers you to uncover meaningful issues early.

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.