Companies everywhere are rushing to stand up an AI Center of Excellence (CoE), viewing it as the essential first step to harnessing the power of artificial intelligence. Yet, a surprising number of these initiatives struggle to deliver tangible business value. They become bogged down in technical debates, produce isolated proofs-of-concept, or fail to align with strategic business needs.
The problem isn't the concept of a CoE, but the execution. Success doesn't come from a standard, tech-focused playbook. The playbook that worked for digital transformation won't work for AI. Success requires unlearning old habits and embracing a new set of strategic truths. This article reveals the six surprising secrets to building an AI CoE that avoids the common pitfalls and becomes a true engine for business transformation.
1. Stop Trying to Write the Perfect AI Strategy
The conventional first step—drafting a comprehensive, long-term AI strategy—is often a mistake. According to Gartner, this approach is like "establishing a battle strategy without knowing if troops have been trained." Before you can chart a five-year course, you must first test your organization's real-world readiness.
Instead, a more tactical, five-step approach that prioritizes quick time-to-value is far more effective:
- Use Cases: Start with the pain points. Find high-impact business problems you can solve fast.
- Skills: Assemble the core talent needed to solve those specific problems.
- Data: Gather only the data required for those targeted use cases.
- Technology: Select the right AI techniques for the problem, skills, and data at hand.
- Organization: After delivering a few proofs-of-concept, determine where the expertise should reside.
Why It Works: This "start small, act fast" method allows an organization to learn from practical application. It reveals your actual strengths and weaknesses in skills, data, and culture before you commit to a rigid strategic plan that might be completely disconnected from operational reality.
2. Place Your AI Hub at the Heart of the Business, Not in IT
One of the most common structural errors is to delegate AI implementation to the IT department. As McKinsey bluntly states, this is often a "recipe for failure." The strategic antidote to this problem is to place your AI hub where it belongs: at the heart of the business.
AI is not just another technology project; it is a fundamental business transformation. To ensure it's treated as such, both Gartner and insights from the University of Technology Sydney (UTS) point to the same core principle: AI governance requires strategic, independent authority. Gartner recommends an "AI lab" independent of both individual Lines of Business (LOB) and IT, reporting to a neutral corporate function.
Critical Insight: By positioning the AI function outside of IT, you ensure it maintains a C-suite-level focus on business value. It signals that AI is a core driver of business transformation, not just an infrastructure upgrade.
3. The Best AI 'Experts' Are Actually Great Managers
The most profound shift in thinking about AI skills is recognizing they aren't purely technical. As talent acquisition expert Glen Cathey highlights from the work of researcher Ethan Mollick, core "AI skills" are fundamentally the skills of a good manager.
This managerial framework involves a clear process:
- Identifying problems that need to be solved.
- Decomposing those problems into smaller, manageable tasks.
- Delegating those tasks to an AI with highly specific instructions, even providing clear examples of "what good looks like."
- Critically reviewing the AI's output and providing constructive feedback for revisions.
The Power of This Insight: Most employees operate as individual contributors, not managers. This explains why so many people struggle to get maximum value from generative AI tools. It reframes the challenge from a niche technical skills gap to a much broader—and more solvable—gap in managerial and critical thinking skills, making true AI proficiency accessible to the entire workforce.
4. Success Isn't Just ROI—It's 'Capability ROI'
While many organizations struggle to demonstrate the value of their AI projects, the most successful ones measure more than just immediate financial returns. According to a model from ISACA, a comprehensive evaluation of AI initiatives requires looking at three distinct categories of return on investment:
Measurable ROI
The direct, quantifiable impacts like cost savings and revenue increases.
Strategic ROI
AI's role in achieving long-term organizational goals (over a 3-to-5-year period), such as gaining a competitive advantage.
Capability ROI
How an AI project improves the organization's overall AI maturity through skills development, creation of new roles, and fostering cultural readiness.
"Every AI project should not only guide a firm towards immediate financial returns but also serve as an investment in the company's capacity to harness AI competitively. Any AI initiative that fails to enhance AI maturity is considered unsuccessful."
— ISACA Report
5. Your CoE Doesn't Have to Be Forever
The prospect of creating a new, permanent organizational structure can be daunting. However, a dedicated AI CoE or governance body can be a transitional arrangement, not necessarily a permanent fixture. According to research from UTS, these dedicated structures are highly effective during the early stages of AI adoption for rapidly building organizational literacy, capability, and maturity.
Framing the CoE as a temporary catalyst is a powerful strategic tool for overcoming organizational inertia:
- Politically: Prevents the creation of a new permanent "empire"
- Financially: Gives it a defined scope and endpoint
- Culturally: Focuses on empowerment, not centralized control
The Ultimate Goal: A transitional CoE isn't meant to work itself out of a job—it acts as a catalyst that transforms federated business units into self-sufficient AI powerhouses, successfully embedding AI governance and skills into "business-as-usual" frameworks over time.
6. Don't Staff Your CoE with Who's Available—Draft Your Top 5%
Picking the right team is the single most critical act of executive sponsorship in a CoE's success. The team cannot be composed of whoever happens to be available. As enterprise AI firm C3 AI recommends, organizations should "Identify the top 5% in the company – only the best architects, developers, and data scientists" to staff their CoE.
Successful CoE team members share a distinct set of attributes:
Why This Matters: This elite staffing philosophy does more than just ensure technical competence; it creates a "gravitational pull" for other high-performers, making the CoE a prestigious rotation. This single decision determines whether the CoE is perceived as a strategic strike team with the credibility to drive real change or a bureaucratic support desk destined to become a sidelined, ineffective function.
Conclusion: From Center of Excellence to Engine of Transformation
The era of treating AI as an IT project is over. The organizations that win the next decade will be those who recognize the AI CoE for what it truly is: not a technical support center, but the strategic engine for rewiring the entire business. Success means looking beyond a traditional playbook and embracing a tactical, business-first approach. It requires empowering the CoE with the right authority, staffing it with your best people, and measuring its success not just in dollars, but in enterprise-wide capability.
As you move forward, ask yourself this:
Is your organization building an AI Center of Excellence, or are you building an engine for future-proofing your entire business?